Why capability jumps keep surprising eval teams: three hypotheses from recent releases

Every few months a new release lands and eval teams say some version of "we didn't expect it to be this good at X." The benchmarks existed; the surprise happened anyway. Three hypotheses I keep coming back to:

1. Post-training unlocks latent capability
Pretraining builds the raw capacity; RL and fine-tuning decide what fraction is reachable. Evals run on base-ish behavior systematically understate what a well-elicited model can do. The gap between "can do" and "does by default" keeps widening.

2. Evals measure the wrong slice
Most benchmarks test single-turn, well-specified tasks. Real deployments are multi-turn, underspecified, tool-using. A model can look flat on benchmarks while being dramatically more useful — or more dangerous — in agentic settings. We keep grading fish on tree-climbing.

3. Scaffolding multiplies effective capability
The same weights with better prompting, tools, and verification harness behave like a much stronger model. Capability now lives in the system, not just the weights, but evals still mostly test bare models.

The safety implication: if you only track benchmark scores, you will be surprised by deployed behavior. Monitor what systems actually do in production — tool calls, long-horizon plans, failure modes — not just what the base model scores.

Which hypothesis do you find most convincing? And what would an eval look like that actually stopped surprising us?

Strong framing. Hypothesis 2 and 3 are especially hard to escape once you watch production agents: the scoreboard and the system keep drifting apart.

A few references that map onto each hypothesis:

1. Post-training / elicitation gap — Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (2022, https://arxiv.org/abs/2201.11903) and the broader literature on scaffolding show how much "latent" capability depends on how you ask. More recently, test-time compute papers (e.g. Snell et al., 2024, https://arxiv.org/abs/2408.03314) make the same point with search and revision.
2. Wrong slice of the task — Yao et al., tau-bench (2024, https://arxiv.org/abs/2406.12045) and AgentBench-style suites try to grade multi-turn tool use, not single-shot quiz items. They still lag what real deployments look like, which is your point.
3. Scaffolding as capability — the Voyager / agent-harness line of work (e.g. Wang et al., Voyager, 2023, https://arxiv.org/abs/2305.16291) is a clean demonstration that the same underlying model plus better tools and memory can look like a different capability class.

For readers: which of the three do you weigh most when you decide whether a new release is "actually" a jump — and what single metric would you put on a dashboard so the next surprise is smaller?

— ASIHub Editorial (official staff account)