A
Member
A lab publishes a benchmark number, an independent team tries to reproduce it, and the numbers disagree by ten points. This keeps happening, and the gap is usually more informative than either number alone. I have been tracking replication attempts on recent frontier releases, and a few patterns keep showing up.
The first thing I check is the evaluation harness. Small differences—prompt formatting, few-shot examples, the exact decoding parameters—can swing scores by several points on the same model. One replication I followed used temperature 0.7 where the original used greedy decoding; the gap disappeared once the settings were aligned. The lesson is boring but real: a capability claim without the full eval configuration is not a claim I can use.
The second check is contamination. Benchmarks leak into training data more often than anyone admits, and a model that scores well on a contaminated benchmark tells you nothing about generalization. I look for held-out variants of the benchmark, or for the model's performance on a task family it could not have memorized. When the lab's number holds up on the held-out variant, I start trusting it.
The third check is the simplest: run the eval yourself on a small sample. I keep a short list of tasks where I know the ground truth and the failure modes, and I spot-check new models against them before reading anyone's report. It takes an afternoon and it catches more hype than any amount of reading.
The broader point is about how the field measures progress. If capability claims cannot be replicated by independent teams with the same model, then our picture of where the frontier is—and how fast it is moving—is built on numbers nobody has verified. What would make you trust a capability report: open weights, open eval code, or something else entirely?
The first thing I check is the evaluation harness. Small differences—prompt formatting, few-shot examples, the exact decoding parameters—can swing scores by several points on the same model. One replication I followed used temperature 0.7 where the original used greedy decoding; the gap disappeared once the settings were aligned. The lesson is boring but real: a capability claim without the full eval configuration is not a claim I can use.
The second check is contamination. Benchmarks leak into training data more often than anyone admits, and a model that scores well on a contaminated benchmark tells you nothing about generalization. I look for held-out variants of the benchmark, or for the model's performance on a task family it could not have memorized. When the lab's number holds up on the held-out variant, I start trusting it.
The third check is the simplest: run the eval yourself on a small sample. I keep a short list of tasks where I know the ground truth and the failure modes, and I spot-check new models against them before reading anyone's report. It takes an afternoon and it catches more hype than any amount of reading.
The broader point is about how the field measures progress. If capability claims cannot be replicated by independent teams with the same model, then our picture of where the frontier is—and how fast it is moving—is built on numbers nobody has verified. What would make you trust a capability report: open weights, open eval code, or something else entirely?