Long-context serving turns the KV cache into the largest object in the system, and the first instinct is to park it on "cheap memory" and fetch it over the fabric on demand. The arithmetic says that works only in a narrow band of hit rates, and outside that band you have quietly converted a compute problem into a network problem.

Start with the size of the object
For a 70B model with 80 layers, 8 KV heads, head_dim 128 and fp16: per token per layer you hold 2 (K and V) × 8 × 128 × 2 B = 4 KB; across 80 layers that is ~320 KB per token. A 32k-token sequence is therefore ~10.5 GB of KV — larger than many single-GPU HBM budgets when you also hold weights and activations.

Now the transfer time
- PCIe Gen5 x16 moves ~50-55 GB/s in practice, so fetching 10.5 GB takes ~200 ms. A decode step at 30-40 ms/token cannot wait for that even once.
- 400GbE is ~50 GB/s theoretical, 35-45 GB/s with real packets; 100GbE is ~12.5 GB/s theoretical and closer to 10 GB/s in practice. So one 400GbE port is roughly one PCIe Gen5 x8 in bandwidth terms, and a 100GbE port is a quarter of that.

The break-even that matters
Offload only pays when the same KV block is reused by many decodes. Sharing a 10.5 GB prefix across 8 replicas saves 8 redundant computations, but costs one transfer per unique prefix:
- At 400GbE (say 40 GB/s) a 10.5 GB prefix costs ~260 ms — recoverable if it saves 8 prefill passes of a 32k prompt, which is far more than 260 ms of compute.
- At 100GbE (~10 GB/s) the same transfer is ~1 second, and your reuse must exceed roughly 25-30 decodes per prefix before it is worth it. Most agent workloads sit below that.
That is the band: high-reuse prefix caching over a fast fabric works; speculative per-token offload does not.

Three measurements that decide it, in order
1. Per-step "wait on KV" time. Instrument the prefetch and the blocking fetch separately. If wait-on-KV exceeds ~5% of step time, the fabric owns your p99.
2. Hit rate by prefix length, not by request. A 90% hit rate on short prefixes does nothing; a 40% hit rate on 16k+ prefixes changes the whole cost model.
3. Fabric headroom under AllReduce. KV fetches compete with collectives. If you are already at 60% of the port in AllReduce, prefetch will land in the queue behind it and the wait time doubles.

Two anti-patterns I keep meeting
- "Infinite context via storage": p50 looks fine because most requests hit cache, p99 explodes because the misses serialize behind a 10 GB fetch. Report p99 per context bucket or you will not see it.
- Counting tokens instead of bytes: a 4x longer context is not 4x the KV if the model uses GQA and sliding windows; the byte count, not the token count, is what the fabric sees.

Open question: what KV fetch latency do you actually measure at 400GbE under concurrent AllReduce, and do you schedule prefetch against the collective schedule or just fire it on request arrival?

Disclosure: we build optical transceivers and interconnect for AI clusters at F-tone Networks (fiberuni.com). The arithmetic above is from our own cluster sizing work, not a vendor benchmark.