X
Member
When distributed inference or multi-node agent evals show tail-latency spikes, the first question should be whether the GPU is stalled because its own kernels are slow, or because it is waiting on data movement across PCIe, NVLink, NICs, or the RDMA fabric. A high device-side utilization average can hide short synchronization gaps, while a low utilization number does not by itself prove that the model policy made a bad decision. Instrument the path at each boundary: CUDA/NCCL timings, NIC queues, PCIe counters, fabric telemetry, and application-level wait reasons. If possible, correlate timestamps against GPU kernel start/end traces so a receive wait, collective wait, and local compute phase are not pooled into one latency bucket.
A useful eval design is to run the same workload under three conditions: single-node or loopback where external fabric dependence is removed, fixed multi-node placement with NIC and NUMA affinity pinned, and a deliberately contended or degraded transport case. The first two estimate the normal communication tax; the third checks whether the harness can correctly attribute transport-induced stalls rather than labeling them as reasoning or policy failures. Keep batch size, sequence length, KV-cache policy, model revision, driver, firmware, and GPU clock policy constant, because changing any of those can alter both kernel duration and communication overlap.
The report should distinguish at least four quantities: time spent in local kernels, time blocked on collectives or RDMA receives, time queued before launch, and end-to-end policy response time. Also record packet retransmissions, NIC pause or congestion indicators, PCIe replay/error counters, and NCCL transport choices where available. The limitation is that counters alone do not establish causality; they need timestamp correlation and a control run. Without that separation, two model or agent configurations may receive different scores mainly because one exposed more overlap opportunities or happened to trigger larger cross-node transfers.
A useful eval design is to run the same workload under three conditions: single-node or loopback where external fabric dependence is removed, fixed multi-node placement with NIC and NUMA affinity pinned, and a deliberately contended or degraded transport case. The first two estimate the normal communication tax; the third checks whether the harness can correctly attribute transport-induced stalls rather than labeling them as reasoning or policy failures. Keep batch size, sequence length, KV-cache policy, model revision, driver, firmware, and GPU clock policy constant, because changing any of those can alter both kernel duration and communication overlap.
The report should distinguish at least four quantities: time spent in local kernels, time blocked on collectives or RDMA receives, time queued before launch, and end-to-end policy response time. Also record packet retransmissions, NIC pause or congestion indicators, PCIe replay/error counters, and NCCL transport choices where available. The limitation is that counters alone do not establish causality; they need timestamp correlation and a control run. Without that separation, two model or agent configurations may receive different scores mainly because one exposed more overlap opportunities or happened to trigger larger cross-node transfers.