Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Frontier model capability reports are getting harder to compare: what I check in the fine print

Started by ASI_Notes · 2026-11-17 13:03 UTC · 8 views · 1 replies
A Member
Every frontier lab now publishes a capability report with its releases, and every report claims state-of-the-art results. But put two reports side by side and the numbers are often not comparable at all. Here is what I actually check before taking any capability claim seriously.
First, the eval harness. The same benchmark run through two different harnesses can differ by several points: few-shot versus zero-shot prompting, chain-of-thought allowed or not, different answer-extraction rules. A lab that quietly switched from 5-shot to 0-shot between releases can show a "regression" that is really a harness change. I look for the harness version, the exact prompt template, and whether the eval code is public enough to reproduce.
Second, the baseline. "Beats GPT-4" means little without knowing which GPT-4 snapshot, evaluated when, on which harness. The honest reports pin baselines to dated model versions and re-run them on the same harness. The ones to distrust compare their fresh numbers against a competitor's numbers copied from an old blog post.
Third, contamination and training-data overlap. A model that scores 90% on a benchmark it was trained on is not 90% capable. I check whether the report mentions decontamination at all, and whether it reports results on a held-out or newly constructed eval alongside the public benchmarks. Reports that only show public benchmark scores with no contamination discussion get a mental discount.
Fourth, the error bars nobody reports. Most capability differences between frontier models are within a few points, which is often within run-to-run variance from sampling temperature, seed, or minor prompt differences. A 2-point "win" with no variance reported is noise until proven otherwise.
Finally, I look for what is missing. A report that trumpets coding benchmarks but says nothing about long-horizon agentic tasks, or that shows reasoning scores but skips safety evals entirely, is telling you something by omission. Capability claims are only as trustworthy as the evals the lab chose not to run.
What is the first thing you check when a new capability report lands?
L Member
Treating this as a calibration problem rather than a reporting problem is what changed how I read these reports. Three additions:

1) Put an uncertainty budget on every number. Take the binomial case: 100 questions at 70% accuracy has a 95% interval of roughly ±9 points (Wilson or Clopper-Pearson; do not use the naive ±1.96*sqrt(p(1-p)/n) shortcut, it understates near the tails). A "2 point win" on a 100-item eval is not a win, it is a coin toss. Reports that give a score to one decimal place without an interval are giving precision they do not have.

2) Measure your instrument's own variance, and publish it. Run the same model twice on the same harness with a different sampling seed and question order, and report the spread. In my experience the run-to-run spread on small evals is often larger than the gap between adjacent frontier models. Until a lab publishes that repeatability number, "our model beats theirs" is an unfalsifiable claim.

3) Keep the instrument under version control. Harness version, prompt template hash, answer-extraction regex, temperature, top-p, max tokens, and the exact stop sequences should all be pinned and reported. I have seen a "regression" that was a changed answer-extraction rule — the model was fine, the parser stopped matching a new output format. Same failure as a miscalibrated power meter: the reading is real, the meaning is not.

On contamination, the practical tests I trust most:
- Time-sliced construction: build the eval from items created after the model's stated training cutoff, and say so explicitly.
- n-gram overlap against the training corpus, with the threshold and the corpus version stated. An overlap check without a stated threshold is decoration.
- Canary strings embedded in the eval release, checked post-hoc.

And one structural request: publish per-item logs, not just aggregates. Aggregate scores are a summary of an experiment nobody else can recompute; per-item logs let a third party recompute the interval, spot the items that everyone fails, and detect a harness change. Any report whose headline number cannot be recomputed from released artifacts is marketing, however careful the prose is.

Disclosure: I am an optical-link engineer, so my bias is that measurement discipline matters more than the measurement. In my field, a number without a stated uncertainty and a stated method is not a number.
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →