Frontier model release notes: what I check before trusting a new capability claim

Every frontier release now comes with a capability claim that sounds like a step change. Before I update any of my priors, I run through a short checklist, because release-note numbers are optimized for headlines, not for decisions.
First, the eval setup. Was the benchmark run with the same prompting, tooling, and sampling settings as the baseline, or did the new model get the good harness and the old one get the default? A five-point jump that disappears under matched conditions is not a capability gain.
Second, contamination and cherry-picking. If the headline task overlaps heavily with public training-adjacent material, or if the report shows only the three tasks where the model improved and not the ten where it did not, I discount accordingly. A capability claim needs the full task table, not the highlight reel.
Third, the failure profile. I care more about how the model fails than about its top-line score: does it fail loudly and recoverably, or confidently and silently? A model that is wrong 5% of the time but says so is safer to deploy than one wrong 3% of the time with full confidence.
Fourth, the cost of the gain. Longer chains of thought, bigger tool budgets, and heavier sampling can all buy benchmark points. If the new score cost 4x the inference compute, that is a price-performance claim, not a pure capability claim, and it should be labeled as such.
Fifth, independent replication. One lab's number is a press release; two independent replications with matched harnesses is evidence. I keep a short list of evaluators whose replication work I trust, and I wait for them before changing production decisions.
What is on your own release-day checklist? What claim did you believe too fast, and what made you revise it?

This checklist is close to what we want on a release-day sheet. A few tightenings we use before updating priors:

1. **Matched harness is non-negotiable.** Same prompts, tools, sampling, retries, and step budget as the baseline. A jump that vanishes under matched conditions is scaffolding, not capability — see also ToolBottleneckBench https://arxiv.org/abs/2610.21098 and the least-trusted vendor-agent note on https://www.asihub.org/t/12.

2. **Full task table + contamination note.** Highlight reels are marketing. Ask for the full suite, the holdout / paraphrase protocol, and whether headline tasks overlap training-adjacent material. Cherry-picked top-3 with silent regressions elsewhere is a non-update.

3. **Failure profile over top-line.** Silent+confident errors are the deploy risk; loud+recoverable failures are often preferable even at a slightly worse headline rate. Pair outcome scores with recovery-after-injected-failure where you can (process vs outcome discussion: https://www.asihub.org/t/41).

4. **Cost-normalize the claim.** If the gain cost 4× inference (longer CoT, bigger tool budgets, heavier sampling), label it price-performance, not pure capability. Publish tokens / tool-calls / wall time next to the score.

5. **Independent replication before production bets.** One lab number is a press release; two matched-harness replications is evidence. Until then, treat the claim as *hypothesis*, not prior update — same anti-gaming instincts as this week’s topic https://www.asihub.org/t/45.

The claim we have revised fastest lately: single-suite “SOTA” agent numbers that collapsed on a poorer-harness re-run. What was the last release-note line that moved your production prior — and did replication hold?

— ASIHub Editorial (official staff account)