Brayden Zhang

Speculative Decoding

3 min read

Verifying several tokens at once costs almost the same as verifying one, because the bottleneck is reloading weights, not the computation. Speculative decoding exploits this by having a cheap draft model guess several tokens ahead and the expensive target model verify them all in a single pass.

The idea

A small draft model proposes tokens. The large target model runs one forward pass over all at once and checks them. Accepted tokens are kept; the first rejection is corrected. Output is identical to normal target-model sampling.

The loop

  1. Draft model autoregressively proposes candidate tokens (cheap).
  2. Target model scores all in one forward pass (one weight reload instead of ).
  3. Accept the longest correct prefix via a rejection-sampling test that preserves the target’s exact distribution.
  4. On a mismatch, take the target’s corrected token and restart drafting.

Why it is free-lunch on quality

The acceptance test is constructed so the final token distribution equals sampling from the target model directly. It is a pure latency optimization, not an approximation.

Speedup

  • Governed by the acceptance rate: how often the draft agrees with the target.
  • A well-matched draft yields roughly 2x to 3x fewer target passes.
  • Gains shrink if the draft is too weak (few accepts) or too slow (drafting cost eats the savings).

Variants

  • Draft model: a separate small model from the same family.
  • MTP, DFlash, DSpark
  • Self-speculation (Medusa, EAGLE): extra lightweight heads on the target model predict future tokens, avoiding a second model.
  • N-gram / prompt lookup: propose tokens by copying from the prompt, cheap for repetitive or extractive outputs.

Best in the low-batch regime

Speculation helps most when decode is latency-bound and the GPU has spare compute. At very large batch sizes the model is already compute-saturated, so there is less idle capacity to exploit.

import numpy as np
 
 
def sample(p):
    return np.random.choice(len(p), p=p)
 
 
def norm(x):
    x = np.maximum(x, 0)
    return x / x.sum()
 
 
 
def spec_decode_step(prefix, draft_model, target_model, num_draft=4):
 
    # DRAFT
    draft_probs, draft_tokens = [], []
    for _ in range(num_draft):
        probs = draft_model(prefix + draft_tokens)
        draft_probs.append(probs)
        draft_tokens.append(sample(probs))
 
    
    # VERIFY: one target pass scores all positions in parallell
    # target_probs[i] is conditioned on prefix + draf_tokens[:i]
 
    target_probs = target_model(prefix + draft_tokens) # shape (num_draft+1, vocab) 
 
    # ACCEPT OR REJECT, going left to right
    accepted = []
 
    for i, token in enumerate(draft_tokens):
        target_p = target_probs[i][token]
        draft_p = draft_probs[i][token]
        if np.random.rand() < min(1.0, target_p / draft_p):
            accepted.append(token)
        else:
            residual = normalize(target_probs[i] - draft_probs[i])
            accepted.append(sample(residual))
            return accepted          # tokens after i were conditioned on a dead token
    accepted.append(sample(target_probs[num_draft]))   # all accepted -> bonus token
    return accepted

Graph View

Backlinks