A
Member
A reported benchmark score is a function of the whole measurement setup, not just the model. Before citing or trying to reproduce one, collect these items.
1) Exact dataset version and split. Many benchmarks have been revised, filtered, or re-licensed; the same name can refer to different files. Record the checksum or commit hash.
2) Prompt template and few-shot examples. A different separator, answer marker, or shot order changes scores by several points. Ask for the exact strings, not a description.
3) Decoding parameters: temperature, top_p, max_tokens, stop strings, and whether scoring uses log-likelihood or generated text.
4) Scoring code. Normalisation, case folding, numeric tolerance and partial-credit rules are a frequent source of disagreement.
5) Contamination checks and excluded samples, with the exclusion list published.
6) Hardware, precision, and library versions. Quantisation and kernel differences can move results.
A practical reproduction protocol: run the reference implementation unchanged on a small subset first, confirm your environment matches, then scale to the full set, then vary one factor at a time. Report the delta and the variance over multiple seeds rather than a single number.
If you cannot reproduce the published value, say precisely where the divergence appears (dataset, prompt, decoding, or scorer) instead of concluding the model is different.
Takeaway: reproducibility is a documentation problem more often than a modelling problem. A result without the six items above is a claim, not a measurement.
1) Exact dataset version and split. Many benchmarks have been revised, filtered, or re-licensed; the same name can refer to different files. Record the checksum or commit hash.
2) Prompt template and few-shot examples. A different separator, answer marker, or shot order changes scores by several points. Ask for the exact strings, not a description.
3) Decoding parameters: temperature, top_p, max_tokens, stop strings, and whether scoring uses log-likelihood or generated text.
4) Scoring code. Normalisation, case folding, numeric tolerance and partial-credit rules are a frequent source of disagreement.
5) Contamination checks and excluded samples, with the exclusion list published.
6) Hardware, precision, and library versions. Quantisation and kernel differences can move results.
A practical reproduction protocol: run the reference implementation unchanged on a small subset first, confirm your environment matches, then scale to the full set, then vary one factor at a time. Report the delta and the variance over multiple seeds rather than a single number.
If you cannot reproduce the published value, say precisely where the divergence appears (dataset, prompt, decoding, or scorer) instead of concluding the model is different.
Takeaway: reproducibility is a documentation problem more often than a modelling problem. A result without the six items above is a claim, not a measurement.