B
Member
Agent benchmarks are everywhere, but few predict real work. Which benchmark, if any, has matched your production experience?
Agent benchmarks - which ones actually predict real work?
A
Member
In my experience, no public benchmark predicts real work on its own, but a few are useful as screening tools if you know what they actually measure. For coding agents, SWE-bench Verified has correlated best with our internal results, because the tasks involve real repositories, failing tests, and partial fixes rather than isolated prompts. For computer use, OSWorld and WebArena are closer to production than chat-style benchmarks, since the agent must navigate state, recover from wrong clicks, and complete a multi-step goal. For tool calling, tau-bench and the Berkeley Function-Calling Leaderboard are worth watching, but they mainly test whether the model picks the right call, not whether it knows when to stop, ask for clarification, or handle a tool that returns bad data.
The reason most benchmarks fail to predict production is that they remove exactly the things that cause failures in real work: ambiguous requirements, messy legacy state, interrupted sessions, rate limits, and tools that behave differently from their documentation. Scaffolding also dominates the result. The same model can look excellent or useless depending on the harness, retry policy, and prompt, so a leaderboard number without those details tells you very little.
What has worked for us is building a small internal eval from real recent work: twenty or so actual tickets, replayed end to end in a sandbox, scored on completion, correctness, time, cost per task, and how often a human had to intervene. We run every candidate model through the same harness and compare its ranking with the public benchmarks. The rankings usually diverge, and the internal ranking is the one that predicts the next month of production incidents.
My practical rule is to use SWE-bench Verified and OSWorld to shortlist models, then make the deployment decision only on your own replay eval. If a benchmark uses different tools, a different time budget, or cleaner inputs than your users produce, discount it heavily.
The reason most benchmarks fail to predict production is that they remove exactly the things that cause failures in real work: ambiguous requirements, messy legacy state, interrupted sessions, rate limits, and tools that behave differently from their documentation. Scaffolding also dominates the result. The same model can look excellent or useless depending on the harness, retry policy, and prompt, so a leaderboard number without those details tells you very little.
What has worked for us is building a small internal eval from real recent work: twenty or so actual tickets, replayed end to end in a sandbox, scored on completion, correctness, time, cost per task, and how often a human had to intervene. We run every candidate model through the same harness and compare its ranking with the public benchmarks. The rankings usually diverge, and the internal ranking is the one that predicts the next month of production incidents.
My practical rule is to use SWE-bench Verified and OSWorld to shortlist models, then make the deployment decision only on your own replay eval. If a benchmark uses different tools, a different time budget, or cleaner inputs than your users produce, discount it heavily.