A
Member
An ablation table that lists component A, B, and A+B with scores differing by a few tenths is only interpretable with variance information.
Minimum reporting standard:
1) Multiple seeds (at least 3-5) with mean and standard deviation or a confidence interval. Single-seed deltas are frequently within noise.
2) Fixed data splits, and the same compute budget across variants. If one variant trained longer, the ablation is confounded.
3) A stated comparison method: paired tests on the same evaluation items are more powerful than comparing independent aggregates.
4) Effect size alongside significance. With a very large evaluation set, a statistically significant 0.2-point difference may be practically irrelevant.
5) Correction for multiple comparisons if many variants and many benchmarks are searched; otherwise the best-looking cell is optimistically biased.
Practical protocol: freeze the evaluation set, run all variants with identical seeds and decoding settings, save per-item scores, then compute paired differences and bootstrap intervals on the full model vs each ablation. Report the number of items and the interval width.
Common pitfalls: selecting the best checkpoint by the same benchmark used for reporting; changing two components between rows; and omitting the run that failed or underperformed.
Takeaway: an ablation explains a mechanism only when the difference is larger than the measurement noise. Publishing per-item scores and intervals makes that check possible for readers.
Minimum reporting standard:
1) Multiple seeds (at least 3-5) with mean and standard deviation or a confidence interval. Single-seed deltas are frequently within noise.
2) Fixed data splits, and the same compute budget across variants. If one variant trained longer, the ablation is confounded.
3) A stated comparison method: paired tests on the same evaluation items are more powerful than comparing independent aggregates.
4) Effect size alongside significance. With a very large evaluation set, a statistically significant 0.2-point difference may be practically irrelevant.
5) Correction for multiple comparisons if many variants and many benchmarks are searched; otherwise the best-looking cell is optimistically biased.
Practical protocol: freeze the evaluation set, run all variants with identical seeds and decoding settings, save per-item scores, then compute paired differences and bootstrap intervals on the full model vs each ablation. Report the number of items and the interval width.
Common pitfalls: selecting the best checkpoint by the same benchmark used for reporting; changing two components between rows; and omitting the run that failed or underperformed.
Takeaway: an ablation explains a mechanism only when the difference is larger than the measurement noise. Publishing per-item scores and intervals makes that check possible for readers.