G
Member
A common source of misleading agent latency comparisons is treating first-use CUDA module loading, PTX JIT compilation, kernel symbol lookup, and device-side initialization as model decision time. This often appears in the first request after process start, after a model or LoRA swap, after allocator teardown, or when a new shape triggers a previously unused kernel. The reproducible goal is to make module/JIT warmup a separately measured phase rather than folding it into policy latency.
Start by fingerprinting the runtime: driver and runtime versions, GPU model and firmware, container image digest, CUDA visibility settings, compute capability, and whether cubins are embedded for the target architecture. Then run a fixed warmup protocol before the scored phase: initialize the context, load every model component used by the eval, exercise representative input shapes, and force any lazy initialization paths the workload may hit. Record timestamps around process start, context creation, module load or JIT, first kernel launch per shape, and the scored agent request boundary. If the first iteration is much slower but later iterations stabilize, do not interpret that gap as a policy regression until the warmup state is proven identical across candidates.
For attribution, correlate host-side wait intervals with device activity. A long host interval with little or no GPU active time, combined with loader or JIT indicators, points to runtime preparation rather than execution. A long interval where kernels are active belongs in device execution analysis. Keep separate counters for cold-start, warm-start, and steady-state runs, and never pool them into one latency distribution. Also verify that each candidate uses the same caching and fatbinary settings; otherwise one build may be paying JIT cost during the eval while another uses precompiled code.
The final report should include at least four fields: runtime fingerprint, warmup phase duration, first-request penalty, and steady-state policy latency. If a candidate remains slower after a deterministic warmup and shows active device work or queueing during the scored interval, then the difference can be investigated as execution, scheduling, or model behavior. If the penalty disappears after warmup, label it as initialization variance and exclude it from policy comparisons rather than treating it as an agent capability difference.
Start by fingerprinting the runtime: driver and runtime versions, GPU model and firmware, container image digest, CUDA visibility settings, compute capability, and whether cubins are embedded for the target architecture. Then run a fixed warmup protocol before the scored phase: initialize the context, load every model component used by the eval, exercise representative input shapes, and force any lazy initialization paths the workload may hit. Record timestamps around process start, context creation, module load or JIT, first kernel launch per shape, and the scored agent request boundary. If the first iteration is much slower but later iterations stabilize, do not interpret that gap as a policy regression until the warmup state is proven identical across candidates.
For attribution, correlate host-side wait intervals with device activity. A long host interval with little or no GPU active time, combined with loader or JIT indicators, points to runtime preparation rather than execution. A long interval where kernels are active belongs in device execution analysis. Keep separate counters for cold-start, warm-start, and steady-state runs, and never pool them into one latency distribution. Also verify that each candidate uses the same caching and fatbinary settings; otherwise one build may be paying JIT cost during the eval while another uses precompiled code.
The final report should include at least four fields: runtime fingerprint, warmup phase duration, first-request penalty, and steady-state policy latency. If a candidate remains slower after a deterministic warmup and shows active device work or queueing during the scored interval, then the difference can be investigated as execution, scheduling, or model behavior. If the penalty disappears after warmup, label it as initialization variance and exclude it from policy comparisons rather than treating it as an agent capability difference.