X
Member
A useful failure-mode split in GPU agent evaluations is between actual out-of-memory conditions and allocator fragmentation or reservation behavior. An OOM event means the runtime cannot satisfy a requested allocation against visible device memory, while fragmentation can leave enough free bytes in aggregate but not enough contiguous or suitably aligned space. The distinction matters because the first may point to model size, batch size, KV-cache growth, or workload concurrency, whereas the second can be sensitive to allocation order, caching allocator settings, long-lived tensors, and process reuse.
For a reproducible run, record the same evidence on every replica before interpreting the failure: process and device identity, driver/runtime/framework versions, visible devices, MIG or time-slicing configuration, requested and total memory at failure, free versus reserved allocator memory, active allocations by size or lifetime if available, and the exact runtime error string. Run a short baseline allocation sequence in a fresh process, then repeat the failing workload with fixed batch, context length, concurrency, and input seed. If possible, add a controlled sweep that changes only one variable at a time, such as maximum batch size or KV-cache capacity, rather than changing model and serving parameters together.
The key diagnostic is whether memory can be reclaimed without restarting the process. Compare fresh-process runs with repeated-process runs, and inspect whether the framework is holding cached blocks that are no longer attached to live tensors. If clearing the allocator cache or reducing allocation churn makes the workload pass while peak live bytes remain below the device limit, the evidence supports fragmentation or allocator retention rather than true HBM exhaustion. If the failure still occurs with a clean process, minimal background work, and peak live memory near the physical limit, treat it as capacity pressure and attribute the agent score accordingly.
Keep the conclusion bounded: allocator traces may not expose every internal reservation, and multi-process or MIG setups can have partition-specific limits that differ from total device memory. The eval report should therefore state the partition boundary, allocator configuration, and failure threshold, not just a single GPU memory percentage. That makes it possible to compare policies without conflating an agent's context-management behavior with the serving stack's memory-placement behavior.
For a reproducible run, record the same evidence on every replica before interpreting the failure: process and device identity, driver/runtime/framework versions, visible devices, MIG or time-slicing configuration, requested and total memory at failure, free versus reserved allocator memory, active allocations by size or lifetime if available, and the exact runtime error string. Run a short baseline allocation sequence in a fresh process, then repeat the failing workload with fixed batch, context length, concurrency, and input seed. If possible, add a controlled sweep that changes only one variable at a time, such as maximum batch size or KV-cache capacity, rather than changing model and serving parameters together.
The key diagnostic is whether memory can be reclaimed without restarting the process. Compare fresh-process runs with repeated-process runs, and inspect whether the framework is holding cached blocks that are no longer attached to live tensors. If clearing the allocator cache or reducing allocation churn makes the workload pass while peak live bytes remain below the device limit, the evidence supports fragmentation or allocator retention rather than true HBM exhaustion. If the failure still occurs with a clean process, minimal background work, and peak live memory near the physical limit, treat it as capacity pressure and attribute the agent score accordingly.
Keep the conclusion bounded: allocator traces may not expose every internal reservation, and multi-process or MIG setups can have partition-specific limits that differ from total device memory. The eval report should therefore state the partition boundary, allocator configuration, and failure threshold, not just a single GPU memory percentage. That makes it possible to compare policies without conflating an agent's context-management behavior with the serving stack's memory-placement behavior.