JustujuAI, layer by layer.Open in the interactive map

Speculative decoding

A small, fast model guesses the next few tokens and the big model checks them all at once: fewer slow steps, with output identical under greedy decoding and the same distribution when sampling.

Layer 6 · Inference & Optimization · Run time · reviewed 2026-10-30

Read first: How a model predicts the next token, Memory & bandwidth (HBM).

D1 · StoryFor young children and total beginners

Predict: if the helper guesses wrong, does the final answer get worse?

A big AI writes one word at a time, and each word is slow.

So a small, quick helper guesses the next few words. The big AI checks them all in one go and keeps the ones it agrees with.

When the helper guesses well, the answer arrives much sooner.

A quick draft, checked by the expert

  1. Quick helper: guesses the next few words
  2. Big AI: checks them all at once
  3. Keep: the words it agrees with

Like: A junior writes a draft and the expert just ticks or corrects it.

D2 · PictureFor curious beginners

Generating each token needs a full read of the big model's weights, and the hardware is mostly waiting on memory. Checking several tokens in one pass costs about the same as generating one.

Speculative decoding uses that spare capacity. A small draft model proposes a few tokens. The big target model scores them all in one pass and keeps the longest run it agrees with, then adds one token of its own.

With the right acceptance rule the output follows exactly the same probabilities as the big model alone. Only the speed changes, and the gain depends on how often the draft is right.

One round of speculative decoding

  1. Draft: small model guesses a few tokens
  2. Check: big model scores all in one pass
  3. Keep: longest run it agrees with
  4. Add one: big model's own next token

Like: A junior writes a draft and the expert just ticks or corrects it.
Where it breaks: An expert editing a draft may let small mistakes through. Here the big model's rule is strict: the final text is distributed exactly as if the big model had written every word itself, so a bad draft only costs time.

D5 · FrontierFor experts

"Same answer, fewer slow steps" needs three qualifiers: the output is identical under greedy decoding, follows the same probability distribution when sampling, and comes faster only while the hardware has spare compute.

Busy servers. Large batches use up the spare compute that made checking cheap, so gains usually shrink. For long contexts, where the KV cache dominates, speed-ups of up to 2.5 times have been shown at batch sizes of 32 to 256 (MagicDec, 2024).

Drafting without a second model. Predicting the big model's internal features rather than tokens gave 2.7 to 3.5 times lower latency on a 70-billion-parameter model while preserving the output distribution (EAGLE, 2024).

Trees instead of lines. Checking a tree of candidate continuations in one pass raises the chance that some branch is accepted, but spends more compute per step (SpecInfer, 2023).

How far to guess. The best number of draft tokens depends on how often the draft is right, which changes as the text gets easier or harder (Leviathan et al., 2022 derive the trade-off).

Examples as of 2026-10-30

Sources

  1. Fast Inference from Transformers via Speculative Decoding (2022)
  2. Accelerating Large Language Model Decoding with Speculative Sampling (2023)
  3. MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding (2024-08-20)
  4. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024-01-26)
  5. SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification (2023-05-16)

How it connects

More in Layer 6 · Inference & Optimization