How a model predicts the next token

A language model reads the text so far, gives every possible next token a probability, picks one, and repeats.

Layer 5 · Foundation Models · Build and run time · Draft, awaiting expert review, reviewed 2026-10-01

Read first: Tokens & tokenization, Transformers & attention.

D1 · StoryFor young children and total beginners

Before you read: what word would you say comes next?

Let's play a game. I say: “The cat sat on the…” and you shout the next word.

You'd probably say “mat”. Maybe “floor” or “sofa”. Probably not “moon”!

An AI language model plays this game again and again, one small piece of a word at a time. That's how it writes whole answers.

Like: The finish-the-sentence game.

D2 · PictureFor curious beginners

Text is split into tokens (words or pieces of words), and each token becomes a list of numbers.

Those numbers pass through a stack of transformer layers. Attention lets each position look back at earlier tokens to decide what matters.

The last layer gives a score to every token in the vocabulary, and softmax turns those scores into probabilities.

One token is chosen, added to the text, and the whole process repeats.

One step of text generation

Tokenize: text to tokens
Embed: tokens to numbers
Transformer: attention + layers
Scores: one per token
Softmax: to probabilities
Pick: and append

Like: Like the autocomplete on your phone, scaled up enormously.
Where it breaks: Phone autocomplete looks at the last few words. A large model attends to its whole context window at once, and its internal computation can represent plans that go beyond the next word.

D3 · PlayFor motivated learners

The model gives every candidate token a score; softmax turns scores into probabilities. Temperature controls how sharp or flat that distribution is before a token is drawn.

Try the tasks below. Watch how the bars change, then press Sample 20 to see what the model would actually write.

Like: Temperature is like how adventurous you feel when ordering: low means your usual favourite, high means you might try anything.
Where it breaks: Your adventurousness has reasons. Temperature just divides every score by the same number before softmax: it changes how random the choice is, not how good or creative the options are.

Try the interactive version in the map.

Things to try

Set the temperature to 0.2. Which word wins almost every time?
Set it to 1.8 and sample again. How often does “moon” appear now?
Limit to the top 3 tokens. Which words can never be chosen?

D4 · MechanismFor practitioners

Each candidate token i gets a score (logit) zᵢ. With temperature T, the probability is pᵢ = exp(zᵢ / T) ÷ Σⱼ exp(zⱼ / T).

Greedy decoding always takes the highest pᵢ. Top-k keeps only the k highest-probability tokens and renormalises. Top-p (nucleus) keeps the smallest set whose probabilities add up to at least p.

Edit the scores and settings below; every column is computed live from the formula.

Try the interactive version in the map.

tokens = tokenize(prompt)
cache = None                       # KV cache: reuse attention work for earlier tokens
while not done(tokens):
    logits, cache = model(tokens[-1:], cache)   # one forward pass for the newest token
    probs = softmax(logits / temperature)
    probs = top_p(top_k(probs, k), p)            # optional filters, then renormalise
    next_token = sample(probs)                   # or argmax(probs) for greedy decoding
    tokens.append(next_token)
return detokenize(tokens)

Trade-offs

Greedy is repeatable but can loop or sound flat; sampling is more varied but less predictable.
Low temperature suits extraction, code and factual answers; higher temperature suits brainstorming and fiction.
Top-p adapts to how confident the model is; top-k uses a fixed cut-off regardless of the distribution's shape.
The KV cache makes each new token cheap to compute but uses memory that grows with context length.

D5 · FrontierFor experts

“Just predicting the next word” undersells what happens inside. Interpretability research has found internal features that represent concepts and, in some cases, evidence of planning ahead, such as choosing a rhyme before writing the line.

Active directions include predicting several tokens at once, speculative decoding, diffusion-style language models that refine a whole sequence, and reinforcement learning that trains models to reason through long chains of tokens before answering.