Decode is memory-bound, where verifying several tokens at once costs almost the same as verifying one, because the bottleneck is reloading weights, not the computation (see Inference Metrics). Speculative decoding exploits this by having a cheap draft model guess several tokens ahead and the expensive target model verify them all in a single pass.
The idea
A small draft model proposes tokens. The large target model runs one forward pass over all at once and checks them. Accepted tokens are kept; the first rejection is corrected. Output is identical to normal target-model sampling.
The loop
- Draft model autoregressively proposes candidate tokens (cheap).
- Target model scores all in one forward pass (one weight reload instead of ).
- Accept the longest correct prefix via a rejection-sampling test that preserves the target’s exact distribution.
- On a mismatch, take the target’s corrected token and restart drafting.
Why it is free-lunch on quality
The acceptance test is constructed so the final token distribution equals sampling from the target model directly. It is a pure latency optimization, not an approximation.
Speedup
- Governed by the acceptance rate: how often the draft agrees with the target.
- A well-matched draft yields roughly 2x to 3x fewer target passes.
- Gains shrink if the draft is too weak (few accepts) or too slow (drafting cost eats the savings).
Variants
- Draft model: a separate small model from the same family.
- Self-speculation (Medusa, EAGLE): extra lightweight heads on the target model predict future tokens, avoiding a second model.
- N-gram / prompt lookup: propose tokens by copying from the prompt, cheap for repetitive or extractive outputs.
Best in the low-batch regime
Speculation helps most when decode is latency-bound and the GPU has spare compute. At very large batch sizes the model is already compute-saturated, so there is less idle capacity to exploit.