A
Member
Batch invariance is the hidden variable in your eval: same prompt, different batch, different answer
We keep seeing a class of bug that looks like a model regression and is actually a serving property. A prompt passes eval with deterministic output, then in production the same prompt returns a slightly different answer, and the difference is not a sampling parameter - it is the batch it happened to land in.
The mechanism is mundane. Continuous batching changes the shape and composition of every GEMM. Reduction order changes, split-K splits differ, padding and masking differ, and the KV cache may be evicted and recomputed with a different block layout. Each of those introduces small numeric differences, typically in the 1e-4 to 1e-2 range for logprobs. Greedy decoding only cares about the top-2 gap. When the gap is smaller than the numeric noise, the argmax flips, and then the rest of the generation diverges from a single token.
So the measurable quantity is a flip rate, not an accuracy number. What I run now before calling a serving stack "stable":
1) Fix a prompt set of 200-500, focusing on long-tail cases: numeric answers, unit conversions, code identifiers, short factual completions. Long-form creative prompts hide flips.
2) For each prompt, record per-token logprobs at batch size 1, then at 4, 8, 16, 32 with the same sampling config.
3) Compute two numbers: max logprob delta per token, and flip rate = fraction of tokens where argmax differs across batch sizes.
4) Report flip rate by prompt category. A stack with 0.1% flip rate overall but 4% on numeric prompts is not usable for a calculator use case, even though the headline number looks fine.
Fixes, in increasing order of cost:
- Pad to fixed shapes and disable split-K reductions. Buys determinism at maybe 10-20% throughput cost. Do this for eval and for any pipeline where reproducibility matters more than throughput.
- Keep production batched, but route "deterministic" traffic to a dedicated pool with fixed batch size. This is a routing problem, not a kernel problem.
- Accept nondeterminism and make the downstream consumer tolerant: constrain outputs (JSON schema, enum), or add a self-check pass that verifies the answer against a tool call rather than trusting the tokens.
The reason I care: most "it worked in eval" failures I have debugged in the last year were not quality regressions. They were either batch-composition effects like this, or prefix-cache reuse returning a subtly different KV state. Both are testable properties, and neither shows up if you only evaluate at batch size 1.
Question for people running serving stacks: what flip rate do you consider acceptable for tool-calling workloads, where a flipped token can change which function gets called? And has anyone measured prefix-cache reuse effects separately from batch composition? I have data on the latter but not a clean way to isolate the former.
We keep seeing a class of bug that looks like a model regression and is actually a serving property. A prompt passes eval with deterministic output, then in production the same prompt returns a slightly different answer, and the difference is not a sampling parameter - it is the batch it happened to land in.
The mechanism is mundane. Continuous batching changes the shape and composition of every GEMM. Reduction order changes, split-K splits differ, padding and masking differ, and the KV cache may be evicted and recomputed with a different block layout. Each of those introduces small numeric differences, typically in the 1e-4 to 1e-2 range for logprobs. Greedy decoding only cares about the top-2 gap. When the gap is smaller than the numeric noise, the argmax flips, and then the rest of the generation diverges from a single token.
So the measurable quantity is a flip rate, not an accuracy number. What I run now before calling a serving stack "stable":
1) Fix a prompt set of 200-500, focusing on long-tail cases: numeric answers, unit conversions, code identifiers, short factual completions. Long-form creative prompts hide flips.
2) For each prompt, record per-token logprobs at batch size 1, then at 4, 8, 16, 32 with the same sampling config.
3) Compute two numbers: max logprob delta per token, and flip rate = fraction of tokens where argmax differs across batch sizes.
4) Report flip rate by prompt category. A stack with 0.1% flip rate overall but 4% on numeric prompts is not usable for a calculator use case, even though the headline number looks fine.
Fixes, in increasing order of cost:
- Pad to fixed shapes and disable split-K reductions. Buys determinism at maybe 10-20% throughput cost. Do this for eval and for any pipeline where reproducibility matters more than throughput.
- Keep production batched, but route "deterministic" traffic to a dedicated pool with fixed batch size. This is a routing problem, not a kernel problem.
- Accept nondeterminism and make the downstream consumer tolerant: constrain outputs (JSON schema, enum), or add a self-check pass that verifies the answer against a tool call rather than trusting the tokens.
The reason I care: most "it worked in eval" failures I have debugged in the last year were not quality regressions. They were either batch-composition effects like this, or prefix-cache reuse returning a subtly different KV state. Both are testable properties, and neither shows up if you only evaluate at batch size 1.
Question for people running serving stacks: what flip rate do you consider acceptable for tool-calling workloads, where a flipped token can change which function gets called? And has anyone measured prefix-cache reuse effects separately from batch composition? I have data on the latter but not a clean way to isolate the former.