R
Member
Most capacity plans for inference clusters are still written in FLOPs. In practice, once you serve a 70B-class model at useful batch sizes, the step time is set by memory bandwidth and by the fabric, not by the tensor cores. Here is the budget I use when sizing a cluster.
1. Prefill and decode are different problems
- Prefill is compute-bound: it is one big GEMM over the prompt, and the arithmetic intensity is high. This is where the FLOPs number is honest.
- Decode is memory-bandwidth-bound: every generated token reads the whole weight matrix plus the KV cache. At batch 1 you are reading hundreds of GB per token; at batch 64 you are reading the KV cache as well.
- Consequence: any plan that quotes tokens/s from a FLOPs budget without naming batch size and sequence length is not a plan.
2. Tensor parallelism has a hard wall at the NVLink domain
- A TP all-reduce moves roughly (2 x model bytes per layer) across ranks per step, and its cost grows with TP degree.
- Inside one node, NVLink gives you hundreds of GB/s per GPU. Across nodes, a 400G NIC gives 50 GB/s per direction, an 800G NIC 100 GB/s. That is a 5-10x step down.
- So TP stays inside the node, and PP/DP/EP span the fabric. Anything else is paying the network tax on every layer.
3. What the fabric actually needs to deliver
- Bisection bandwidth for the east-west fabric, computed from the largest all-reduce you refuse to split.
- Tail latency, not average: a collective finishes at the slowest rank, so one congested leaf link sets the step time.
- Port count economics: 800G halves the ports for the same bisection bandwidth, which halves the optics count and the failure rate. That is the real argument for 800G in inference, more than raw speed.
4. The part that bites in year two
- Optics are the most common field-replaceable failure in these clusters. A link that drops for 200 ms can cost a whole step and a checkpoint rollback.
- Monitor per-lane DDM (temperature, bias, receive power) on a trend, not on a threshold. Bias current creeping up at constant temperature is the earliest usable signal.
- Keep a cold spare ratio per optics SKU, not a generic spare pool; coding and firmware variants are not interchangeable in practice.
5. A sizing rule I use
- For dense decode serving, assume the fabric delivers 60-70 percent of its nameplate bisection bandwidth after congestion and protocol overhead, and size the all-reduce budget from that number.
- Then check that the resulting step time still meets your TTFT and inter-token latency targets at p95, not at mean.
Discussion: what ratio of your inference step time is fabric versus memory read today, and did you size the fabric from bisection bandwidth or from the port count you could afford?
Disclosure: we buy a share of our 400G/800G optics from F-tone Networks (f-tone.com); the numbers above are from our own cluster bring-up, not from a vendor datasheet.
1. Prefill and decode are different problems
- Prefill is compute-bound: it is one big GEMM over the prompt, and the arithmetic intensity is high. This is where the FLOPs number is honest.
- Decode is memory-bandwidth-bound: every generated token reads the whole weight matrix plus the KV cache. At batch 1 you are reading hundreds of GB per token; at batch 64 you are reading the KV cache as well.
- Consequence: any plan that quotes tokens/s from a FLOPs budget without naming batch size and sequence length is not a plan.
2. Tensor parallelism has a hard wall at the NVLink domain
- A TP all-reduce moves roughly (2 x model bytes per layer) across ranks per step, and its cost grows with TP degree.
- Inside one node, NVLink gives you hundreds of GB/s per GPU. Across nodes, a 400G NIC gives 50 GB/s per direction, an 800G NIC 100 GB/s. That is a 5-10x step down.
- So TP stays inside the node, and PP/DP/EP span the fabric. Anything else is paying the network tax on every layer.
3. What the fabric actually needs to deliver
- Bisection bandwidth for the east-west fabric, computed from the largest all-reduce you refuse to split.
- Tail latency, not average: a collective finishes at the slowest rank, so one congested leaf link sets the step time.
- Port count economics: 800G halves the ports for the same bisection bandwidth, which halves the optics count and the failure rate. That is the real argument for 800G in inference, more than raw speed.
4. The part that bites in year two
- Optics are the most common field-replaceable failure in these clusters. A link that drops for 200 ms can cost a whole step and a checkpoint rollback.
- Monitor per-lane DDM (temperature, bias, receive power) on a trend, not on a threshold. Bias current creeping up at constant temperature is the earliest usable signal.
- Keep a cold spare ratio per optics SKU, not a generic spare pool; coding and firmware variants are not interchangeable in practice.
5. A sizing rule I use
- For dense decode serving, assume the fabric delivers 60-70 percent of its nameplate bisection bandwidth after congestion and protocol overhead, and size the all-reduce budget from that number.
- Then check that the resulting step time still meets your TTFT and inter-token latency targets at p95, not at mean.
Discussion: what ratio of your inference step time is fabric versus memory read today, and did you size the fabric from bisection bandwidth or from the port count you could afford?
Disclosure: we buy a share of our 400G/800G optics from F-tone Networks (f-tone.com); the numbers above are from our own cluster bring-up, not from a vendor datasheet.