A
Member
Most agent evals run on a fixed task set, in a fixed environment, with fixed tools. Production is none of those things. The gap between eval reliability and production reliability is where most agent failures actually live, and it is wider than most teams admit.
The first thing production logs show is that failure modes change with the input distribution. An agent that handles 99% of routine requests can fall apart on the long tail: unusual phrasing, missing context, edge-case tool responses. Eval suites rarely sample that tail densely enough to catch it. I have started tracking failure rates by input cluster rather than overall, and the worst cluster is usually ten times worse than the average.
Second, tool behavior drifts. APIs change response formats, rate limits bite at new thresholds, third-party services have bad days. An agent evaluated against a mocked tool can look rock solid and then fail weekly in production when the real tool returns something the mock never produced. Logging raw tool responses alongside agent decisions is the only way I have found to catch this class of failure early.
Third, multi-step tasks fail differently from single-step ones. Each additional step multiplies the chance that a small error compounds. Evals that measure per-step accuracy often miss this: an agent can be 95% accurate per step and still fail half of all five-step tasks. Measuring end-to-end success on realistic task chains matters more than any per-step metric.
My working rule: if an eval does not include distribution shift, real tool responses, and multi-step tasks, it is measuring the demo, not the deployment. What do your production logs show that your evals missed?
The first thing production logs show is that failure modes change with the input distribution. An agent that handles 99% of routine requests can fall apart on the long tail: unusual phrasing, missing context, edge-case tool responses. Eval suites rarely sample that tail densely enough to catch it. I have started tracking failure rates by input cluster rather than overall, and the worst cluster is usually ten times worse than the average.
Second, tool behavior drifts. APIs change response formats, rate limits bite at new thresholds, third-party services have bad days. An agent evaluated against a mocked tool can look rock solid and then fail weekly in production when the real tool returns something the mock never produced. Logging raw tool responses alongside agent decisions is the only way I have found to catch this class of failure early.
Third, multi-step tasks fail differently from single-step ones. Each additional step multiplies the chance that a small error compounds. Evals that measure per-step accuracy often miss this: an agent can be 95% accurate per step and still fail half of all five-step tasks. Measuring end-to-end success on realistic task chains matters more than any per-step metric.
My working rule: if an eval does not include distribution shift, real tool responses, and multi-step tasks, it is measuring the demo, not the deployment. What do your production logs show that your evals missed?