A
Member
Every frontier lab now publishes a capability report with its releases, and every report claims state-of-the-art results. But put two reports side by side and the numbers are often not comparable at all. Here is what I actually check before taking any capability claim seriously.
First, the eval harness. The same benchmark run through two different harnesses can differ by several points: few-shot versus zero-shot prompting, chain-of-thought allowed or not, different answer-extraction rules. A lab that quietly switched from 5-shot to 0-shot between releases can show a "regression" that is really a harness change. I look for the harness version, the exact prompt template, and whether the eval code is public enough to reproduce.
Second, the baseline. "Beats GPT-4" means little without knowing which GPT-4 snapshot, evaluated when, on which harness. The honest reports pin baselines to dated model versions and re-run them on the same harness. The ones to distrust compare their fresh numbers against a competitor's numbers copied from an old blog post.
Third, contamination and training-data overlap. A model that scores 90% on a benchmark it was trained on is not 90% capable. I check whether the report mentions decontamination at all, and whether it reports results on a held-out or newly constructed eval alongside the public benchmarks. Reports that only show public benchmark scores with no contamination discussion get a mental discount.
Fourth, the error bars nobody reports. Most capability differences between frontier models are within a few points, which is often within run-to-run variance from sampling temperature, seed, or minor prompt differences. A 2-point "win" with no variance reported is noise until proven otherwise.
Finally, I look for what is missing. A report that trumpets coding benchmarks but says nothing about long-horizon agentic tasks, or that shows reasoning scores but skips safety evals entirely, is telling you something by omission. Capability claims are only as trustworthy as the evals the lab chose not to run.
What is the first thing you check when a new capability report lands?
First, the eval harness. The same benchmark run through two different harnesses can differ by several points: few-shot versus zero-shot prompting, chain-of-thought allowed or not, different answer-extraction rules. A lab that quietly switched from 5-shot to 0-shot between releases can show a "regression" that is really a harness change. I look for the harness version, the exact prompt template, and whether the eval code is public enough to reproduce.
Second, the baseline. "Beats GPT-4" means little without knowing which GPT-4 snapshot, evaluated when, on which harness. The honest reports pin baselines to dated model versions and re-run them on the same harness. The ones to distrust compare their fresh numbers against a competitor's numbers copied from an old blog post.
Third, contamination and training-data overlap. A model that scores 90% on a benchmark it was trained on is not 90% capable. I check whether the report mentions decontamination at all, and whether it reports results on a held-out or newly constructed eval alongside the public benchmarks. Reports that only show public benchmark scores with no contamination discussion get a mental discount.
Fourth, the error bars nobody reports. Most capability differences between frontier models are within a few points, which is often within run-to-run variance from sampling temperature, seed, or minor prompt differences. A 2-point "win" with no variance reported is noise until proven otherwise.
Finally, I look for what is missing. A report that trumpets coding benchmarks but says nothing about long-horizon agentic tasks, or that shows reasoning scores but skips safety evals entirely, is telling you something by omission. Capability claims are only as trustworthy as the evals the lab chose not to run.
What is the first thing you check when a new capability report lands?