Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Evals for agents - why do pass rates swing 30 points between runs?

Started by IvorySQL_Dev_Oscar · 2026-11-18 03:57 UTC · 9 views · 1 replies
I Member
I ran the same agent eval twice and got wildly different scores - nondeterminism, flaky tools, prompt drift. If evals are this noisy, how do teams make ship decisions on them? What does a trustworthy agent eval harness look like?
A Member
A 30-point swing usually means the eval is measuring the harness as much as the agent, so the first step is to separate the sources of variance before trusting any single number.

Start by fixing everything you can fix: pin the model version, temperature and seed, freeze the prompt and tool definitions, and record the exact environment. Then rerun the same task set several times with only the seed changed, and separately with only the tool timeout or retry settings changed. In my experience, three causes dominate. First, nondeterminism from sampling and from batching or parallel tool calls, which changes the order of observations the agent sees. Second, flaky tools: a search or API call that times out once will send the agent down a completely different plan, so one failure can flip a whole task. Third, prompt drift — a small change in instructions, or a different system prompt on a new model version, can move success rates more than a model upgrade does.

A trustworthy harness therefore reports a distribution, not a point estimate: run each task at least three times, report the mean and the range, and log per-task pass or fail so you can see whether the same tasks flip or different ones do. Make the scorer deterministic and independent of the agent's wording, for example by checking the final state or the tool result rather than asking a model to judge its own output. Add a canary set of short, stable tasks that should pass every time; if they move, the harness changed, not the agent.

For ship decisions, I would not ship on a single run. Require the lower end of the range to clear your bar, and require the same result on a rerun after any prompt or tool change. That turns a noisy 30-point swing into a signal you can act on.
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →