Ad广告位招商 / Advertise hereSlot A1 · Rate card →

Inference clusters are becoming interconnect-bound - the topology budget nobody writes down

Started by RayPeng · 2026-11-17 10:46 UTC · 6 views · 1 replies
R Member
Most capacity plans for inference clusters are still written in FLOPs. In practice, once you serve a 70B-class model at useful batch sizes, the step time is set by memory bandwidth and by the fabric, not by the tensor cores. Here is the budget I use when sizing a cluster.

1. Prefill and decode are different problems
- Prefill is compute-bound: it is one big GEMM over the prompt, and the arithmetic intensity is high. This is where the FLOPs number is honest.
- Decode is memory-bandwidth-bound: every generated token reads the whole weight matrix plus the KV cache. At batch 1 you are reading hundreds of GB per token; at batch 64 you are reading the KV cache as well.
- Consequence: any plan that quotes tokens/s from a FLOPs budget without naming batch size and sequence length is not a plan.

2. Tensor parallelism has a hard wall at the NVLink domain
- A TP all-reduce moves roughly (2 x model bytes per layer) across ranks per step, and its cost grows with TP degree.
- Inside one node, NVLink gives you hundreds of GB/s per GPU. Across nodes, a 400G NIC gives 50 GB/s per direction, an 800G NIC 100 GB/s. That is a 5-10x step down.
- So TP stays inside the node, and PP/DP/EP span the fabric. Anything else is paying the network tax on every layer.

3. What the fabric actually needs to deliver
- Bisection bandwidth for the east-west fabric, computed from the largest all-reduce you refuse to split.
- Tail latency, not average: a collective finishes at the slowest rank, so one congested leaf link sets the step time.
- Port count economics: 800G halves the ports for the same bisection bandwidth, which halves the optics count and the failure rate. That is the real argument for 800G in inference, more than raw speed.

4. The part that bites in year two
- Optics are the most common field-replaceable failure in these clusters. A link that drops for 200 ms can cost a whole step and a checkpoint rollback.
- Monitor per-lane DDM (temperature, bias, receive power) on a trend, not on a threshold. Bias current creeping up at constant temperature is the earliest usable signal.
- Keep a cold spare ratio per optics SKU, not a generic spare pool; coding and firmware variants are not interchangeable in practice.

5. A sizing rule I use
- For dense decode serving, assume the fabric delivers 60-70 percent of its nameplate bisection bandwidth after congestion and protocol overhead, and size the all-reduce budget from that number.
- Then check that the resulting step time still meets your TTFT and inter-token latency targets at p95, not at mean.

Discussion: what ratio of your inference step time is fabric versus memory read today, and did you size the fabric from bisection bandwidth or from the port count you could afford?

Disclosure: we buy a share of our 400G/800G optics from F-tone Networks (f-tone.com); the numbers above are from our own cluster bring-up, not from a vendor datasheet.
A Member
This matches what I have seen in practice, and the prefill versus decode split is the right place to start. Prefill can hide a weak fabric behind large, compute-dense operations, while decode exposes it on every token, because each step has to read the weights and the key-value state and then synchronise across ranks before the next token can start. A cluster that looks fine on aggregate throughput can therefore still miss inter-token latency targets badly.

On your discussion question, the fabric share of step time depends strongly on batch size and on how widely the model is split. At very small batches in decode, memory reads dominate, often around three quarters of step time, with the fabric taking only a small share. As batch size grows and the model is split across more devices, the collective per layer grows with it, and the fabric share can rise to roughly a quarter to a third of step time, with tail effects making it worse. The average is misleading here: a collective finishes at the slowest rank, so one congested link or one slow device sets the step time for everyone, which is why p95 and p99 matter far more than the mean.

I would also size from bisection bandwidth first and treat port count as a budget constraint, not as the sizing target. The correct sequence is to work out the largest collective you refuse to slow down, the step-time budget implied by your latency targets, and the bisection bandwidth that collective needs, then derive how many ports that implies at a given link speed. Sizing from the ports you can afford reverses that logic and usually produces an oversubscribed leaf-spine design where the nominal bandwidth exists on paper but not under simultaneous collectives. Your 60 to 70 percent derating is a reasonable starting point; under contention I would plan closer to the lower end for latency-sensitive decode, and verify it with a synthetic collective load rather than with inference traffic alone.

Your point about optics monitoring is also worth stressing. Trending per-lane receive power, bias current and error counts over time catches degradation weeks before a hard failure, whereas a fixed threshold only fires once a link is already dropping. Keeping spares per transceiver variant, and separating prefill and decode pools where possible so their very different fabric demands do not interfere, are both practices I would add to the sizing rule you propose.
Ad广告位招商 / Advertise hereSlot A10 · Rate card →
Log in or register to reply.
Ad广告位招商 / Advertise hereSlot A9 · Rate card →