When deploying large language models for inference, the biggest bottleneck is often GPU memory — not compute. Specifically, the Key-Value (KV) cache can easily consume more memory than the model itself, especially for long-context workloads.

## Why KV Cache Is So Expensive

During autoregressive generation, every token's Key and Value tensors are cached to avoid recomputation. The memory cost grows linearly with context length:

KV Cache memory = 2 × num_layers × seq_len × hidden_dim × dtype_bytes

For a 7B model (32 layers, 4096 hidden, FP16):
- Single sequence, 4k tokens: 2.1 GB
- 100 concurrent sequences: 210 GB

That's more than the model weights themselves.

## The Problem With Traditional Approaches

**Static allocation (PyTorch native):**
Each request gets pre-allocated memory for its maximum length. But most requests never use the full budget. Real utilization is often 20-40%.

**Dynamic allocation:**
On-demand allocation sounds good, but frequent malloc/free causes severe memory fragmentation. Long-sequence workloads fragment the GPU memory so badly that you end up with less usable memory than static allocation.

## How PagedAttention Fixes This

vLLM's insight was to borrow a trick from operating systems: virtual memory and paging.

**The idea:**
1. Divide the KV cache into fixed-size blocks (e.g., 16 tokens per block)
2. Blocks don't need to be physically contiguous
3. Each request maintains a block table that maps logical blocks to physical blocks

This is exactly how OS virtual memory works — the process sees contiguous virtual addresses, but physical memory is scattered across page frames, with a page table doing the translation.

## Real-World Impact

From vLLM's published benchmarks:
- Throughput: 2-4x faster than HuggingFace Transformers
- Memory utilization: from 20-40% up to 80-90%
- Up to 24x higher throughput than naive HuggingFace deployment

## Our Production Experience

We deploy a 7B model on 8× A100:
- Model weights: 14 GB (FP16)
- Remaining GPU memory all goes to KV cache
- Concurrent requests: from ~15 to ~60
- Latency: P50 50ms, P99 200ms

## Practical Tips

1. **Choose the right block size** — too small = block table overhead; too large = internal fragmentation. 16-32 tokens per block works well.

2. **Continuous batching matters** — traditional batching waits for all requests to finish. Continuous batching dynamically adjusts the batch every token generation step. This alone can double throughput.

3. **Prefix cache reuse** — if multiple requests share the same system prompt, cache the prefix KV blocks. Hugely beneficial for multi-turn conversations.

What KV cache optimization strategies are you using in production?