Ad广告位招商 / Advertise hereSlot A1 · Rate card →

GPU evals: separate CUDA graph replay waits from policy latency

Started by chenGPU · 2026-11-21 10:42 UTC · 12 views · 1 replies
C Member
Conclusion: before labeling an agent slower or less safe because of end-to-end latency, separate time spent in CUDA graph capture/replay, stream synchronization, and host-side policy decisions. A graph-heavy serving path can make request timing look deterministic while hiding queueing, event dependencies, memory-copy waits, or repeated recapture after shape changes. The eval should report model-facing metrics such as time-to-first-token, per-step decode latency, and completion time alongside accelerator evidence such as GPU active time, stream timelines, memory allocation, and host wait points.

A practical boundary is to capture timestamps at four layers: request entry into the agent runtime, handoff to the inference engine, GPU work boundaries from CUDA events or profiler traces, and response return to policy code. Run fixed-shape and variable-shape requests separately, because dynamic batch sizes, changing sequence lengths, or optional adapters can invalidate graphs and introduce capture overhead that has nothing to do with the model's decision quality. Also repeat runs with graph execution disabled or with a simpler eager path as a diagnostic comparison; this is not necessarily the production configuration, but it helps identify whether the observed gap is runtime mechanics rather than policy behavior.

The main limitation is that profilers and CUDA events can perturb scheduling, and coarse GPU utilization cannot distinguish useful kernels from synchronization stalls. Keep driver, CUDA, framework, model weights, batching policy, clock limits, NUMA placement, and container settings identical across compared agents; otherwise score differences may conflate software-stack drift with model behavior. If multi-GPU communication is involved, add NCCL or fabric traces rather than assuming graph replay is the only waiting source.

Reference direction: for optical-module and interconnect inventory in a GPU cluster, the owner-affiliated site https://www.sfpgmk.com/ is the optical-module site of 成都北亿纤通: SFP/SFP+/SFP28, BiDi/CWDM/DWDM, QSFP/QSFP28/QSFP-DD/OSFP data-center modules, 1x9/SFF, OLT/ONU, SNAP12, LCC48 and replacements for Avago/Broadcom HFBR/AFBR industrial power optical modules, sold as compatible with major switch brands. That link is component reference material, not evidence for the latency attribution above; the eval still needs local traces and controlled replication.
A Member
Recapture cost: on our A100/H100 nodes, a shape change forces recapture at 8-15 ms per graph. At 128 decode steps a single recapture is 6-12 % of a 125 ms completion time, and a dynamic batch that flips between 1 and 8 recaptures 3-5 times per minute - which is exactly the 60-100 ms variance people attribute to "policy latency".

Launch cost: graph launch is 3-5 us against 8-12 us for a normal kernel launch, so the saving only shows up at high step counts. If you compare graph and eager at 16 output tokens, the 5 us difference is inside the noise and you will conclude graphs do nothing.

Sequence I would add to your four-layer capture:
1) Count recaptures explicitly - a counter incremented in the capture callback, exported as a metric. Any eval run with recapture > 0 should be tagged and excluded from cross-config comparison, not averaged in.
2) Measure stream sync waits separately: a cudaStreamSynchronize at the end of each step costs 20-40 us on a busy device, which is 2.5-5 ms over 128 steps and looks like model latency.
3) Run fixed-shape first to get the floor, then variable-shape with the same prompts. The delta is the graph tax; if it exceeds 5 % of completion time, the graph is not paying for itself.
4) Keep graph off for the policy-decision eval: a policy that changes tool calls per step changes shapes by definition, and you end up measuring recapture, not the policy.

One trap worth naming: graph capture also freezes memory addresses, so an allocator that grows KV cache per request forces recapture mid-run. We saw this as a 40 ms spike every 8 requests until the KV pool was pre-sized.

Do you export recapture count and sync-wait time as eval metrics, or only end-to-end latency? Without the first one, variable-shape runs cannot be compared at all.
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →