JustujuAI, layer by layer.Open in the interactive map

How a model predicts the next token

A language model reads the text so far, gives every possible next token a probability, picks one, and repeats.

Layer 5 · Foundation Models · Build and run time · reviewed 2026-10-30

Read first: Tokens & tokenization, Transformers & attention.

D1 · StoryFor young children and total beginners

Before you read: what word would you say comes next?

Let's play a game. I say: “The cat sat on the…” and you shout the next word.

You'd probably say “mat”. Maybe “floor” or “sofa”. Probably not “moon”!

An AI language model plays this game again and again, one small piece of a word at a time. That's how it writes whole answers.

Like: The finish-the-sentence game.

D2 · PictureFor curious beginners

Text is split into tokens (words or pieces of words), and each token becomes a list of numbers.

Those numbers pass through a stack of transformer layers. Attention lets each position look back at earlier tokens to decide what matters.

The last layer gives a score to every token in the vocabulary, and softmax turns those scores into probabilities.

One token is chosen, added to the text, and the whole process repeats.

One step of text generation

  1. Tokenize: text to tokens
  2. Embed: tokens to numbers
  3. Transformer: attention + layers
  4. Scores: one per token
  5. Softmax: to probabilities
  6. Pick: and append

Like: Like the autocomplete on your phone, scaled up enormously.
Where it breaks: Phone autocomplete looks at the last few words. A large model attends to its whole context window at once, and its internal computation can represent plans that go beyond the next word.

D3 · PlayFor motivated learners

The model gives every candidate token a score; softmax turns scores into probabilities. Temperature controls how sharp or flat that distribution is before a token is drawn.

Try the tasks below. Watch how the bars change, then press Sample 20 to see what the model would actually write.

Like: Temperature is like how adventurous you feel when ordering: low means your usual favourite, high means you might try anything.
Where it breaks: Your adventurousness has reasons. Temperature just divides every score by the same number before softmax: it changes how random the choice is, not how good or creative the options are.

Try the interactive version in the map.

Things to try

D4 · MechanismFor practitioners

Each candidate token i gets a score (logit) zᵢ. With temperature T, the probability is pᵢ = exp(zᵢ / T) ÷ Σⱼ exp(zⱼ / T).

Greedy decoding always takes the highest pᵢ. Top-k keeps only the k highest-probability tokens and renormalises. Top-p (nucleus) keeps the smallest set whose probabilities add up to at least p.

Edit the scores and settings below; every column is computed live from the formula.

Try the interactive version in the map.

tokens = tokenize(prompt)
cache = None                       # KV cache: reuse attention work for earlier tokens
while not done(tokens):
    logits, cache = model(tokens[-1:], cache)   # one forward pass for the newest token
    probs = softmax(logits / temperature)
    probs = top_p(top_k(probs, k), p)            # optional filters, then renormalise
    next_token = sample(probs)                   # or argmax(probs) for greedy decoding
    tokens.append(next_token)
return detokenize(tokens)

Trade-offs

D5 · FrontierFor experts

D2's loop of predict, pick, repeat is accurate, but it undersells what happens inside and hides real limits of the training objective.

Planning inside. Interpretability work found a model picking a rhyme word before writing the line that leads to it, though the method gave clear insight on only about a quarter of prompts tried (Lindsey et al., 2025).

Teacher forcing can fail. On a simple path-finding task, next-token training failed to learn the task at all, while predicting several tokens ahead helped (Bachmann and Nagarajan, 2024).

Several tokens at once. Training to predict four tokens ahead improved coding results at 13B and allowed up to three times faster inference (Gloeckle et al., 2024).

Not left to right. An 8B diffusion language model trained from scratch was competitive with similar autoregressive models, so left-to-right order is not the only option (Nie et al., 2025).

Examples as of 2026-10-30

Sources

  1. Attention Is All You Need (2017)
  2. On the Biology of a Large Language Model (2025)
  3. Better & Faster Large Language Models via Multi-token Prediction (2024)
  4. Fast Inference from Transformers via Speculative Decoding (2022)
  5. The pitfalls of next-token prediction (Bachmann and Nagarajan) (2024-03-11)
  6. Large Language Diffusion Models (Nie et al.) (2025-02-14)

How it connects

More in Layer 5 · Foundation Models