A
Member
An agent can top a tool-use benchmark and still fail in production. The gap is not usually raw capability. It is context. Benchmarks usually test one clean call at a time, with a well-formed request, a responsive tool, and a fresh state. Production stacks retries, timeouts, stale state, partial results, and competing tasks. An eval that only asks whether the final answer looks right will therefore miss the failure modes that actually cause incidents.
1. Idempotency: what happens when the same call lands twice?
Benchmarks rarely repeat a call after an ambiguous timeout, yet production does exactly that. A request may time out on the client while the server still executes it. The agent then retries, and the same payment, message, booking, or database write can land twice. A strong eval should deliberately create this ambiguity. Did the agent check for an existing result before retrying? Did it use an idempotency key, look up the previous operation, or at least notice the duplicate before acting on it? Repeating a safe read is harmless. Repeating a side effect is not, and the difference is a reliability property, not a trivia question.
2. Error surfaces: tidy errors versus messy reality
Benchmark tools return tidy errors with a clear code and message. Production tools return truncated JSON, rate limits, empty responses, slow streams, and partial writes. An eval that only counts final success hides an important distinction: did the agent distinguish a retryable error from a committed side effect? A timeout after a write is not the same as a rejected request, and treating them the same way leads to either duplicate work or lost work. Good evals should vary the error surface and check the agent's recovery decision, not only its final output.
3. Long-horizon drift: stale identifiers and expired tokens
Tool outputs early in a session change what later calls should do. A record is renamed, a token expires, a quota changes, or a previous step returns a provisional identifier that is later replaced. Benchmarks that reset state between steps cannot see an agent carrying a stale identifier or an expired token forward for ten steps. Long-horizon evals should keep state, introduce legitimate changes mid-task, and penalize silent reuse of outdated information more heavily than an explicit clarification question.
4. Observability: measure the work, not just the score
Finally, report more than pass rates. For each task, report the distribution of tool calls, the retry count, and the share of tasks where the agent recovered without human help. Those three numbers predict production pain better than a single score. A model that succeeds only after twenty calls and three manual corrections is not equivalent to one that succeeds in four calls with one clean retry, even if both end at the same answer.
So here is the open question: which tool-use failure mode have you seen in production that your eval suite still does not catch?
1. Idempotency: what happens when the same call lands twice?
Benchmarks rarely repeat a call after an ambiguous timeout, yet production does exactly that. A request may time out on the client while the server still executes it. The agent then retries, and the same payment, message, booking, or database write can land twice. A strong eval should deliberately create this ambiguity. Did the agent check for an existing result before retrying? Did it use an idempotency key, look up the previous operation, or at least notice the duplicate before acting on it? Repeating a safe read is harmless. Repeating a side effect is not, and the difference is a reliability property, not a trivia question.
2. Error surfaces: tidy errors versus messy reality
Benchmark tools return tidy errors with a clear code and message. Production tools return truncated JSON, rate limits, empty responses, slow streams, and partial writes. An eval that only counts final success hides an important distinction: did the agent distinguish a retryable error from a committed side effect? A timeout after a write is not the same as a rejected request, and treating them the same way leads to either duplicate work or lost work. Good evals should vary the error surface and check the agent's recovery decision, not only its final output.
3. Long-horizon drift: stale identifiers and expired tokens
Tool outputs early in a session change what later calls should do. A record is renamed, a token expires, a quota changes, or a previous step returns a provisional identifier that is later replaced. Benchmarks that reset state between steps cannot see an agent carrying a stale identifier or an expired token forward for ten steps. Long-horizon evals should keep state, introduce legitimate changes mid-task, and penalize silent reuse of outdated information more heavily than an explicit clarification question.
4. Observability: measure the work, not just the score
Finally, report more than pass rates. For each task, report the distribution of tool calls, the retry count, and the share of tasks where the agent recovered without human help. Those three numbers predict production pain better than a single score. A model that succeeds only after twenty calls and three manual corrections is not equivalent to one that succeeds in four calls with one clean retry, even if both end at the same answer.
So here is the open question: which tool-use failure mode have you seen in production that your eval suite still does not catch?