Conclusion: before comparing agent policies, treat cold-start GPU costs as a distinct latency population, not as part of steady-state prefill or decode. Context creation, module load, JIT compilation, allocator pool growth, and vendor-library autotuning can dominate the first few requests and then vanish. If a harness averages them into the measured window, a policy that changes request ordering or batching can appear better or worse for reasons unrelated to its decisions. Report warmup separately or exclude it explicitly, and state which choice was made.

A practical diagnostic is to record a per-request timeline with monotonic host timestamps for process start, context creation complete, first real kernel launch, first device synchronize that covers real work, and a marker around policy decision versus model invocation. Run the same fixed request script with a warmup phase and a measured phase in one process, then repeat with a fresh process per trial. If the variance collapses when the process is reused but reappears when each trial starts fresh, the dominant term is process or context initialization, not policy. Also verify that the policy-evaluation clock starts only after warmup is complete; harness wall-clock totals often include tokenizer load, client setup, and loopback transport, which are separate confounders. Where device-side timestamps are available, correlate them with the host markers rather than trusting a single elapsed-time field.

Limitations matter here. Removing warmup does not prove the remaining latency is policy-caused; it only removes a known confounder. The measured phase still assumes stable clocks, no background ECC or Xid events, and no fabric contention, so those need their own telemetry. JIT caching across processes changes the cold-start shape: a populated on-disk cache can make a fresh process look much faster than a truly cold one, so record whether the cache was warm. On MIG or time-sliced devices, context initialization may include scheduling waits that are not attributable to the model, and warmup requests may themselves consume shared capacity. These are reasons to keep cold-start in a separate bucket, not to discard it silently.

Reference direction for eval reports: publish cold-start p50 and p95 separately from steady-state p50 and p95, state how many requests were discarded as warmup, note whether the JIT cache and allocator pool were cold or warm, and say whether the process was reused across trials. If cold start is genuinely part of the user-visible scenario, keep it as its own score instead of blending it into the policy comparison. The useful rule is simple: first show that the measured latency population is stationary and that the policy clock covers only the intended interval, then interpret differences between policies.