Speculative decoding
A small, fast model guesses the next few tokens and the big model checks them all at once, giving the same output in fewer slow steps.
Read first: How a model predicts the next token, Memory & bandwidth (HBM).
D1 · StoryFor young children and total beginners
A big AI writes one word at a time, and each word is slow.
So a small, quick helper guesses the next few words. The big AI checks them all in one go and keeps the ones it agrees with.
When the helper guesses well, the answer arrives much sooner.
Like: A junior writes a draft and the expert just ticks or corrects it.
D2 · PictureFor curious beginners
Generating each token needs a full read of the big model's weights, and the hardware is mostly waiting on memory. Checking several tokens in one pass costs about the same as generating one.
Speculative decoding uses that spare capacity. A small draft model proposes a few tokens. The big target model scores them all in one pass and keeps the longest run it agrees with, then adds one token of its own.
With the right acceptance rule the output follows exactly the same probabilities as the big model alone. Only the speed changes, and the gain depends on how often the draft is right.
One round of speculative decoding
- Draft: small model guesses a few tokens
- Check: big model scores all in one pass
- Keep: longest run it agrees with
- Add one: big model's own next token
Like: A junior writes a draft and the expert just ticks or corrects it.
Where it breaks: An expert editing a draft may let small mistakes through. Here the big model's rule is strict: the final text is distributed exactly as if the big model had written every word itself, so a bad draft only costs time.
D5 · FrontierFor experts
The gain shrinks when a server is already busy with large batches, because the spare compute that made checking cheap is no longer spare. That is why it shines for single users and local machines.
Open work includes drafting without a separate model (extra prediction heads, or reusing the big model's own early layers), drafting trees of alternatives instead of one line, and choosing how many tokens to guess as the text gets easier or harder.