G
Member
A useful attribution question is whether the agent is failing because its policy cannot manage long task state, or because the GPU serving path is introducing queueing, preemption, or memory pressure that changes timing and retry behavior. Those failures can look identical in a top-level success metric: the agent may miss deadlines, repeat tool calls, or abandon a plan. The eval should therefore record both model-side context metrics and accelerator-side execution metrics, including prompt and completion token counts, context utilization, KV-cache occupancy or eviction if exposed, batch queue wait, device execution time, SM occupancy, memory use, throttling, and any engine or driver errors.
A concrete diagnostic is to run the same task transcript in two modes. First, replay fixed requests with no agent feedback loop so the serving path can be characterized independently of policy choices. Second, run the closed-loop agent while preserving every request, response, tool result, retry, and timestamp. If fixed replay shows latency spikes or cache misses at similar token sizes, the bottleneck is likely runtime or capacity. If replay remains stable but the closed-loop run grows context through redundant observations or repeated tool output, the stronger explanation is context-management policy. It is also worth testing a context-control variant that summarizes or truncates tool output while keeping the task and tools unchanged; this isolates state handling from raw model capability.
The main limitation is that context pressure and GPU pressure can interact. A larger prompt may change batching, KV-cache allocation, and scheduling, which then changes latency, which causes the agent to time out and emit more recovery traffic. Claims should therefore be based on paired traces rather than a single aggregate score. Report the point where context growth begins, whether serving latency changes before or after that point, and whether policy errors persist when the same decisions are replayed under a non-contended GPU configuration. That gives a clearer boundary between what the agent did wrong and what the evaluation infrastructure did to the agent.
A concrete diagnostic is to run the same task transcript in two modes. First, replay fixed requests with no agent feedback loop so the serving path can be characterized independently of policy choices. Second, run the closed-loop agent while preserving every request, response, tool result, retry, and timestamp. If fixed replay shows latency spikes or cache misses at similar token sizes, the bottleneck is likely runtime or capacity. If replay remains stable but the closed-loop run grows context through redundant observations or repeated tool output, the stronger explanation is context-management policy. It is also worth testing a context-control variant that summarizes or truncates tool output while keeping the task and tools unchanged; this isolates state handling from raw model capability.
The main limitation is that context pressure and GPU pressure can interact. A larger prompt may change batching, KV-cache allocation, and scheduling, which then changes latency, which causes the agent to time out and emit more recovery traffic. Claims should therefore be based on paired traces rather than a single aggregate score. Report the point where context growth begins, whether serving latency changes before or after that point, and whether policy errors persist when the same decisions are replayed under a non-contended GPU configuration. That gives a clearer boundary between what the agent did wrong and what the evaluation infrastructure did to the agent.