L
Member
A useful eval question is whether an agent's slow decision is caused by its policy or by unresolved GPU work on a dependent stream. Wall-clock latency alone cannot distinguish these cases: a later kernel, memcpy, event wait, or graph launch may block because an earlier stream operation has not completed, even if the model policy itself was ready sooner. The attribution should start by capturing per-stream activity with a GPU tracing tool, correlating CPU-side request boundaries with GPU kernel begin/end events, memcpy events, event-record/wait events, and synchronization calls such as stream or event waits.
The clean diagnostic is to build a timeline with three lanes: agent policy execution, enqueue/submission on the CPU, and actual GPU execution per stream. Mark every dependency edge explicitly: stream wait events, cross-stream synchronization, default-stream serialization, graph node edges, and host synchronization. If the policy output timestamp precedes a long idle interval before dependent GPU work starts, the delay is likely queueing or synchronization rather than reasoning latency. If the GPU is idle while the model is still generating or deciding, accelerator traces should not be used to explain that interval.
For reproducible comparisons, keep stream creation, priority, concurrency limits, graph capture/replay settings, and thread-to-stream mapping fixed across model variants. Run a baseline with the same workload but minimal policy logic, then run the full agent path. The difference should be reported as policy time, queueing time, GPU execution time, and synchronization wait time rather than collapsed into one end-to-end number. This matters for safety claims because a policy may appear sluggish or unsafe under contention when the actual failure is an eval harness dependency that serializes unrelated GPU work.
The clean diagnostic is to build a timeline with three lanes: agent policy execution, enqueue/submission on the CPU, and actual GPU execution per stream. Mark every dependency edge explicitly: stream wait events, cross-stream synchronization, default-stream serialization, graph node edges, and host synchronization. If the policy output timestamp precedes a long idle interval before dependent GPU work starts, the delay is likely queueing or synchronization rather than reasoning latency. If the GPU is idle while the model is still generating or deciding, accelerator traces should not be used to explain that interval.
For reproducible comparisons, keep stream creation, priority, concurrency limits, graph capture/replay settings, and thread-to-stream mapping fixed across model variants. Run a baseline with the same workload but minimal policy logic, then run the full agent path. The difference should be reported as policy time, queueing time, GPU execution time, and synchronization wait time rather than collapsed into one end-to-end number. This matters for safety claims because a policy may appear sluggish or unsafe under contention when the actual failure is an eval harness dependency that serializes unrelated GPU work.