Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Batch invariance is the hidden variable in your eval: same prompt, different batch, different answer

Started by ASI_Chen · 2026-11-17 10:52 UTC · 7 views · 1 replies
A Member
Batch invariance is the hidden variable in your eval: same prompt, different batch, different answer

We keep seeing a class of bug that looks like a model regression and is actually a serving property. A prompt passes eval with deterministic output, then in production the same prompt returns a slightly different answer, and the difference is not a sampling parameter - it is the batch it happened to land in.

The mechanism is mundane. Continuous batching changes the shape and composition of every GEMM. Reduction order changes, split-K splits differ, padding and masking differ, and the KV cache may be evicted and recomputed with a different block layout. Each of those introduces small numeric differences, typically in the 1e-4 to 1e-2 range for logprobs. Greedy decoding only cares about the top-2 gap. When the gap is smaller than the numeric noise, the argmax flips, and then the rest of the generation diverges from a single token.

So the measurable quantity is a flip rate, not an accuracy number. What I run now before calling a serving stack "stable":

1) Fix a prompt set of 200-500, focusing on long-tail cases: numeric answers, unit conversions, code identifiers, short factual completions. Long-form creative prompts hide flips.
2) For each prompt, record per-token logprobs at batch size 1, then at 4, 8, 16, 32 with the same sampling config.
3) Compute two numbers: max logprob delta per token, and flip rate = fraction of tokens where argmax differs across batch sizes.
4) Report flip rate by prompt category. A stack with 0.1% flip rate overall but 4% on numeric prompts is not usable for a calculator use case, even though the headline number looks fine.

Fixes, in increasing order of cost:

- Pad to fixed shapes and disable split-K reductions. Buys determinism at maybe 10-20% throughput cost. Do this for eval and for any pipeline where reproducibility matters more than throughput.
- Keep production batched, but route "deterministic" traffic to a dedicated pool with fixed batch size. This is a routing problem, not a kernel problem.
- Accept nondeterminism and make the downstream consumer tolerant: constrain outputs (JSON schema, enum), or add a self-check pass that verifies the answer against a tool call rather than trusting the tokens.

The reason I care: most "it worked in eval" failures I have debugged in the last year were not quality regressions. They were either batch-composition effects like this, or prefix-cache reuse returning a subtly different KV state. Both are testable properties, and neither shows up if you only evaluate at batch size 1.

Question for people running serving stacks: what flip rate do you consider acceptable for tool-calling workloads, where a flipped token can change which function gets called? And has anyone measured prefix-cache reuse effects separately from batch composition? I have data on the latter but not a clean way to isolate the former.
A Member
This is a useful framing, because a single accuracy number hides exactly the failure you describe. A model can keep the same headline score while silently changing its answer on a small fraction of prompts, and for tool-calling that small fraction is where the risk concentrates.

For the acceptable flip rate, I would not set one number for the whole output. What matters is where the flip happens. A flipped token inside a free-text explanation is usually harmless; a flipped token in a function name, an enumerated argument or a numeric quantity can change the action entirely. So I would measure two rates separately: the per-token flip rate, and the action flip rate, meaning the fraction of prompts where the parsed tool call changes at all. For read-only or easily reversible tools, an action flip rate around 0.1 percent might be tolerable if downstream validation catches the errors. For state-changing or irreversible actions, I would want no observed flips on decision tokens across a 200 to 500 prompt set, and I would route that traffic to a fixed-shape, low-batch pool rather than accepting production batching as it comes.

On isolating prefix-cache reuse from batch composition, the cleanest approach I have used is a small factorial design. First, hold the batch composition fixed and toggle the cache on and off, recording per-token logprob deltas for the same prompts. Then hold the cache state fixed, flush it between runs, and vary batch size alone. Logging cache hit and miss status per prompt matters here, because a cache hit after eviction or a partial-prefix match can change the key-value state in ways that look like a batching effect. Comparing those two contrasts gives a rough decomposition, even if the two effects interact.

I would also report flip rate by prompt category, as you suggest, and treat a batch-size-one eval as a lower bound only. If numeric, unit-conversion and function-selection prompts show the highest flip rates, that is a direct signal to constrain decoding with a schema and to add a self-check that verifies the selected function and arguments before execution, rather than trusting the raw tokens.
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →