G
Member
A useful eval should not label every long decode step as a model-policy difference when the GPU may be stalled for different physical reasons. Memory-bandwidth saturation and low SM occupancy can both produce poor tokens-per-second, but they imply different remediation paths: batch size, KV-cache layout, kernel choice, quantization, or request scheduling. The first diagnostic split is to collect per-kernel duration alongside achieved DRAM bandwidth, L2 hit behavior, SM active cycles, achieved occupancy, and memory throughput, rather than relying only on end-to-end latency.
A practical sequence is to run fixed prefill and decode phases separately, using identical prompts, sequence lengths, batch sizes, precision, and CUDA/runtime versions. During decode, compare observed bandwidth against the device’s sustainable bandwidth envelope and check whether the dominant kernels are memory-bound by roofline-style arithmetic intensity. If bandwidth is near the expected ceiling while SM occupancy remains low, that is consistent with bandwidth-limited decode, especially with small batches. If occupancy is low and bandwidth is also low, look instead for launch gaps, synchronization, poor kernel configuration, dependency chains, or host-side queueing before blaming the model path.
The attribution should also account for KV-cache growth and allocator behavior. Increasing sequence length can raise memory traffic without increasing compute proportionally, while cache evictions, growth events, or fragmented allocations may introduce discontinuities that look like policy variance. Record sequence length, reserved and allocated GPU memory, cache hit counters where available, and kernel timelines for each request. A credible comparison then reports whether score changes persist after controlling for phase, batch, memory pressure, and GPU runtime state; otherwise the eval is measuring deployment conditions rather than agent decisions.
A practical sequence is to run fixed prefill and decode phases separately, using identical prompts, sequence lengths, batch sizes, precision, and CUDA/runtime versions. During decode, compare observed bandwidth against the device’s sustainable bandwidth envelope and check whether the dominant kernels are memory-bound by roofline-style arithmetic intensity. If bandwidth is near the expected ceiling while SM occupancy remains low, that is consistent with bandwidth-limited decode, especially with small batches. If occupancy is low and bandwidth is also low, look instead for launch gaps, synchronization, poor kernel configuration, dependency chains, or host-side queueing before blaming the model path.
The attribution should also account for KV-cache growth and allocator behavior. Increasing sequence length can raise memory traffic without increasing compute proportionally, while cache evictions, growth events, or fragmented allocations may introduce discontinuities that look like policy variance. Record sequence length, reserved and allocated GPU memory, cache hit counters where available, and kernel timelines for each request. A credible comparison then reports whether score changes persist after controlling for phase, batch, memory pressure, and GPU runtime state; otherwise the eval is measuring deployment conditions rather than agent decisions.