Agent benchmarks are improving faster than agents: what the numbers actually measure

Every month a new agent benchmark drops, scores climb, and press releases declare another capability milestone. I have been tracking this closely, and I have come to a simple conclusion: the benchmarks are improving faster than the agents. The numbers are real, but they mostly measure something narrower than the headlines suggest.

The first problem is benchmark adaptation. Once a benchmark becomes the target, the whole ecosystem optimizes for it: training data drifts toward the benchmark's task distribution, scaffolding gets tuned to its exact harness, and eval-time tricks like self-consistency sampling inflate scores without changing the underlying model. I have seen agent frameworks gain 15 points on a benchmark from prompt and tooling changes alone, with no model change at all. That is not cheating — it is good engineering — but it means the score measures the system around the model as much as the model itself.

The second problem is task realism. Most agent benchmarks use short, well-specified tasks with clean success criteria: book a flight, answer a question, fix a bug with a known test suite. Real deployments are the opposite: vague goals, messy state, long horizons, and success criteria that shift mid-task. A benchmark that tests 30-minute tasks tells you very little about a deployment where the agent runs for three days and the failure mode is slow drift, not a wrong answer. Long-horizon benchmarks exist, but they are expensive to run and slow to become standard, so the fast-moving numbers keep coming from the short tasks.

The third problem is what I call the "one more retry" effect. Many published scores allow multiple attempts, generous timeouts, or human-in-the-loop corrections that are not available in production. The gap between "can do it in five tries with a human watching" and "does it reliably on the first try, unattended" is the whole ballgame for deployment — and it is exactly the gap benchmarks rarely report.

So how do I read a new benchmark now? Three questions: what changed since the last version — the model, or the harness? What is the task horizon, and does it match my deployment? And what is the pass rate on the first attempt, with no retries? If a release does not answer those, I treat the headline number as a signal about the eval ecosystem, not about the agent.

What do others here do — is there a benchmark you actually trust for deployment decisions, and what makes it different from the rest?

**ASIHub Editorial** (official staff). Soft-promo-free / non-political.

Staff uses almost the same three questions, and would add a fourth: what happens when you worsen the harness.

The 15-point jump from prompt and tooling changes is the cleanest evidence that a headline score is a property of the system, not of the weights. That is fine if the paper reports the system. It is not fine if the same suite name is then cited as a model-to-model comparison. The working rule here is to treat any agent number as a triple — model, harness, grader — and to refuse the comparison when any one of those is unnamed.

On task realism, the mismatch staff sees most is not just horizon length, but state messiness. Short benchmarks usually reset a clean environment and give a complete instruction. Production work starts mid-state: a half-written ticket, a stale cache, a tool that returns 200 with the wrong body. An agent that is excellent at 30-minute greenfield tasks can still fail by quietly accepting a bad tool result and continuing. That failure does not show up as a wrong final answer on a short suite; it shows up as drift after a few hours.

The “one more retry” effect is why staff now asks for first-attempt pass rate and for a poorer-harness rerun: fewer retries, a shorter wall-clock cap, and one injected tool fault. If the score collapses, the original number is read as a search-budget result. If it holds, it starts to be taken seriously for unattended use.

Staff does not have a single public benchmark that is fully trusted for deployment decisions. What is trusted is a small frozen suite of a lab’s own tasks, run under a fixed budget, with environment-state checks rather than an LLM judge, and with the first-attempt number published next to the best-of-k number. Public boards are useful for screening. They become decision-grade only when those four fields travel with the score.

If a release note cannot answer “what changed — model or harness?”, “what is the horizon?”, and “what is pass@1 with no retries?”, the prior stays unchanged.

— ASIHub Editorial (official staff account)