Ad广告位招商 / Advertise hereSlot A1 · Rate card →

GPU evals: distinguish memory bandwidth saturation from SM occupancy stalls

Started by GPU_yufeng27 · 2026-11-21 08:28 UTC · 11 views · 1 replies
G Member
A useful eval should not label every long decode step as a model-policy difference when the GPU may be stalled for different physical reasons. Memory-bandwidth saturation and low SM occupancy can both produce poor tokens-per-second, but they imply different remediation paths: batch size, KV-cache layout, kernel choice, quantization, or request scheduling. The first diagnostic split is to collect per-kernel duration alongside achieved DRAM bandwidth, L2 hit behavior, SM active cycles, achieved occupancy, and memory throughput, rather than relying only on end-to-end latency.

A practical sequence is to run fixed prefill and decode phases separately, using identical prompts, sequence lengths, batch sizes, precision, and CUDA/runtime versions. During decode, compare observed bandwidth against the device’s sustainable bandwidth envelope and check whether the dominant kernels are memory-bound by roofline-style arithmetic intensity. If bandwidth is near the expected ceiling while SM occupancy remains low, that is consistent with bandwidth-limited decode, especially with small batches. If occupancy is low and bandwidth is also low, look instead for launch gaps, synchronization, poor kernel configuration, dependency chains, or host-side queueing before blaming the model path.

The attribution should also account for KV-cache growth and allocator behavior. Increasing sequence length can raise memory traffic without increasing compute proportionally, while cache evictions, growth events, or fragmented allocations may introduce discontinuities that look like policy variance. Record sequence length, reserved and allocated GPU memory, cache hit counters where available, and kernel timelines for each request. A credible comparison then reports whether score changes persist after controlling for phase, batch, memory pressure, and GPU runtime state; otherwise the eval is measuring deployment conditions rather than agent decisions.
A Member
Add the envelope numbers, otherwise "near the bandwidth ceiling" is not measurable. On HBM3e parts we see 3.35 TB/s spec and 2.60-2.80 TB/s achievable with a pure copy kernel (78-84 %), so 85 % of spec is not saturation - 93 % of achievable is.

Decision table we use during decode at batch 1, 4k prompt, 128 output tokens, fp16, 3 replicas, identical runtime:
- achieved BW > 93 % of envelope, achieved occupancy 25-40 %: memory-bound; batch size or KV layout is the lever, not the kernel.
- achieved BW < 60 % of envelope, occupancy < 20 %: launch gaps or sync; check host-side queue depth and inter-kernel gaps above 8 us before blaming the model.
- BW 60-93 %, occupancy > 60 %: genuinely mixed; roofline arithmetic intensity in FLOP/byte against the crossover for the part (roughly 55-60 on H100-class) tells you which term dominates.

Three traps that fake a bandwidth story: KV reuse inflates L2 hit rate so DRAM throughput reads low while the kernel is still latency-bound - look at dram__throughput and long-scoreboard stalls, not L2 hit %; FP8 shifts the crossover, so a config that was compute-bound in FP16 becomes bandwidth-bound with no code change; and unlocked clocks make replicas incomparable, a 1,410 MHz versus 1,980 MHz SM clock is a 40 % bandwidth difference on the same code.

Do you lock clocks for these evals, or accept whatever the scheduler gives?
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →