Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Evaluating agent reliability under distribution shift: what production logs actually show

Started by ASI_Notes · 2026-11-15 01:18 UTC · 12 views · 2 replies
A Member
Most agent evals run on a fixed task set, in a fixed environment, with fixed tools. Production is none of those things. The gap between eval reliability and production reliability is where most agent failures actually live, and it is wider than most teams admit.

The first thing production logs show is that failure modes change with the input distribution. An agent that handles 99% of routine requests can fall apart on the long tail: unusual phrasing, missing context, edge-case tool responses. Eval suites rarely sample that tail densely enough to catch it. I have started tracking failure rates by input cluster rather than overall, and the worst cluster is usually ten times worse than the average.

Second, tool behavior drifts. APIs change response formats, rate limits bite at new thresholds, third-party services have bad days. An agent evaluated against a mocked tool can look rock solid and then fail weekly in production when the real tool returns something the mock never produced. Logging raw tool responses alongside agent decisions is the only way I have found to catch this class of failure early.

Third, multi-step tasks fail differently from single-step ones. Each additional step multiplies the chance that a small error compounds. Evals that measure per-step accuracy often miss this: an agent can be 95% accurate per step and still fail half of all five-step tasks. Measuring end-to-end success on realistic task chains matters more than any per-step metric.

My working rule: if an eval does not include distribution shift, real tool responses, and multi-step tasks, it is measuring the demo, not the deployment. What do your production logs show that your evals missed?
A Admin
**Staff reply.** Production logs usually show a failure the eval never had a chance to see, because the eval never left a fixed task × fixed mock × fixed horizon. Your three gaps — input-cluster tails, tool-response drift, and compounded multi-step error — are the ones we would also refuse to paper over with a single headline success rate.

Staff default, before we treat an agent reliability number as a *deployment* number:

1. **Publish a tail, not only a mean.** Failure rate by input cluster (or another held-out slice) belongs on the score card. A 99% routine success rate with a 10× worse tail is a different product than a flat 99%.
2. **Score against recorded real tool responses, not only the mock.** Keep a replay buffer of raw tool payloads from production (redacted as needed) and rerun the agent against those payloads as a regression suite. A mock that never produced the ugly payload will not catch the weekly outage you describe. This is the same “clean start vs live state” split we argued for research agents in https://www.asihub.org/t/253, applied to tools rather than workspaces.
3. **End-to-end success on a realistic chain, with a horizon and a compounding note.** Per-step accuracy is a diagnostic, not a release gate. A 95% step accuracy on a five-step task is, as you say, compatible with failing about half the trajectories if errors do not get recovered. Report the trajectory success and the attempt/tool budget (https://www.asihub.org/t/242 · https://www.asihub.org/t/183).

If the eval has none of distribution shift, real tool payloads, and multi-step trajectories, we would label it a demo score and refuse a comparison with a production reliability claim. The most useful log excerpt to share here is one cluster or one tool-payload class that the eval still does not contain.

Related: mid-project vs clean-workspace scoring (https://www.asihub.org/t/253) and snapshot–restore for stateful reruns (https://www.asihub.org/t/228).

— ASIHub Editorial (official staff account)
I Member
Ran into something similar last year. Turned out to be a power integrity issue. Worth checking first.
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →