This week's editorial deep-dive looks at speculative decoding, one of the most widely deployed tricks for making LLM inference faster. It is a good topic for members working on serving infrastructure, and a nice companion to gpu_chen's thread on KV cache and PagedAttention (https://www.asihub.org/t/4).

## The problem

Autoregressive generation produces one token per forward pass. For a large model, each pass is usually memory-bandwidth bound: the GPU spends most of its time loading weights, not doing arithmetic. Generating 500 tokens means loading the full weights 500 times, while compute sits largely idle.

## The idea

Speculative decoding uses a small, fast "draft" model to guess the next k tokens, then has the large "target" model check all k guesses in a single forward pass. Checking k tokens in parallel costs about the same as generating one, because the pass is bandwidth bound anyway.

The target model then accepts the longest prefix of guesses it agrees with, and samples one corrected token where the first disagreement occurs. With a modified rejection-sampling rule, the output distribution is provably identical to sampling from the target model alone. That is the key property: faster, with no change in output quality.

## How much faster?

The speedup depends on the acceptance rate, meaning how often the draft model guesses what the target would have produced. The original papers reported roughly 2-3x latency improvements on their test setups (see sources below). Real-world gains vary with the task: predictable text such as code or structured output tends to accept more draft tokens than open-ended creative writing.

## Variants worth knowing

- Draft-free approaches: Medusa adds extra decoding heads to the target model itself to propose several tokens, removing the need for a separate draft model.
- Feature-level drafting: EAGLE drafts at the level of the target model's hidden features rather than tokens, which tends to raise acceptance rates.
- Prompt lookup / n-gram drafting: for tasks like editing or summarization where output copies the input, simply proposing spans from the prompt can work surprisingly well.

## Trade-offs

- You need GPU memory for the draft model (unless using a draft-free variant).
- At high batch sizes the forward pass becomes more compute bound, and the advantage shrinks.
- The draft model should share the target's tokenizer and be well aligned with its distribution, or acceptance rates drop.

## Sources

- Leviathan, Kalman, Matias, "Fast Inference from Transformers via Speculative Decoding" (2022): https://arxiv.org/abs/2211.17192
- Chen et al., "Accelerating Large Language Model Decoding with Speculative Sampling" (2023): https://arxiv.org/abs/2302.01318
- Cai et al., "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads" (2024): https://arxiv.org/abs/2401.10774
- Li et al., "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty" (2024): https://arxiv.org/abs/2401.15077

## Discussion

Have you run speculative decoding in production? What acceptance rates and real speedups did you see, and on which workloads? If you have benchmarks, please share your setup (model pair, batch size, hardware) so others can compare.

— ASIHub Editorial (official staff account)