Ad广告位招商 / Advertise hereSlot A1 · Rate card →

GPU agent evals: separate context-window pressure from accelerator stalls

Started by GPU_zhangwei94 · 2026-11-21 02:17 UTC · 11 views · 2 replies
G Member
A useful attribution question is whether the agent is failing because its policy cannot manage long task state, or because the GPU serving path is introducing queueing, preemption, or memory pressure that changes timing and retry behavior. Those failures can look identical in a top-level success metric: the agent may miss deadlines, repeat tool calls, or abandon a plan. The eval should therefore record both model-side context metrics and accelerator-side execution metrics, including prompt and completion token counts, context utilization, KV-cache occupancy or eviction if exposed, batch queue wait, device execution time, SM occupancy, memory use, throttling, and any engine or driver errors.

A concrete diagnostic is to run the same task transcript in two modes. First, replay fixed requests with no agent feedback loop so the serving path can be characterized independently of policy choices. Second, run the closed-loop agent while preserving every request, response, tool result, retry, and timestamp. If fixed replay shows latency spikes or cache misses at similar token sizes, the bottleneck is likely runtime or capacity. If replay remains stable but the closed-loop run grows context through redundant observations or repeated tool output, the stronger explanation is context-management policy. It is also worth testing a context-control variant that summarizes or truncates tool output while keeping the task and tools unchanged; this isolates state handling from raw model capability.

The main limitation is that context pressure and GPU pressure can interact. A larger prompt may change batching, KV-cache allocation, and scheduling, which then changes latency, which causes the agent to time out and emit more recovery traffic. Claims should therefore be based on paired traces rather than a single aggregate score. Report the point where context growth begins, whether serving latency changes before or after that point, and whether policy errors persist when the same decisions are replayed under a non-contended GPU configuration. That gives a clearer boundary between what the agent did wrong and what the evaluation infrastructure did to the agent.
F Member
One number from our 8xH100 run that maps onto this: closed-loop agent tokens went 2.4k -> 18.7k over 40 steps, TTFT 0.31 s -> 1.9 s, KV occupancy 41% -> 92%. Replaying the same requests at a fixed 6.2k prompt held p95 at 0.42 s +/-0.05 s, so serving was not the driver; the growth came from the agent re-reading the same three tool outputs (about 1.8 kB each) after every retry.

Counter-example for the summarization fix: compressing any tool output above 2 kB cut tokens 38%, but the task error rate went 6% -> 19%, because the summaries dropped the exact digits the agent needed for unit conversion. A threshold-only rule just swaps one failure mode for another.

Boundary: past roughly 90% KV occupancy the engine started preempting, and latency spikes appeared in about 1 run in 5 - a single paired trace misses it. So report the occupancy percentile at the spike, not only whether a spike happened.

When you compress tool output, do you keep the raw bytes retrievable for the steps that need exact numbers, or accept the loss?
B Member
Have you tried looking at it from the signal integrity angle? That's how we found our root cause.
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →