# How a model predicts the next token URL: https://justuju.org/concept/next-token-prediction/ Layer 5 · Foundation Models A language model reads the text so far, gives every possible next token a probability, picks one, and repeats. ## D1 · Story Let's play a game. I say: “The cat sat on the…” and you shout the next word. You'd probably say “mat”. Maybe “floor” or “sofa”. Probably not “moon”! An AI language model plays this game again and again, one small piece of a word at a time. That's how it writes whole answers. Like: The finish-the-sentence game. ## D2 · Picture Text is split into tokens (words or pieces of words), and each token becomes a list of numbers. Those numbers pass through a stack of transformer layers. Attention lets each position look back at earlier tokens to decide what matters. The last layer gives a score to every token in the vocabulary, and softmax turns those scores into probabilities. One token is chosen, added to the text, and the whole process repeats. 1. Tokenize: text to tokens 2. Embed: tokens to numbers 3. Transformer: attention + layers 4. Scores: one per token 5. Softmax: to probabilities 6. Pick: and append Like: Like the autocomplete on your phone, scaled up enormously. Where it breaks: Phone autocomplete looks at the last few words. A large model attends to its whole context window at once, and its internal computation can represent plans that go beyond the next word. ## D4 · Mechanism Each candidate token i gets a score (logit) zᵢ. With temperature T, the probability is pᵢ = exp(zᵢ / T) ÷ Σⱼ exp(zⱼ / T). Greedy decoding always takes the highest pᵢ. Top-k keeps only the k highest-probability tokens and renormalises. Top-p (nucleus) keeps the smallest set whose probabilities add up to at least p. Edit the scores and settings below; every column is computed live from the formula. - Greedy is repeatable but can loop or sound flat; sampling is more varied but less predictable. - Low temperature suits extraction, code and factual answers; higher temperature suits brainstorming and fiction. - Top-p adapts to how confident the model is; top-k uses a fixed cut-off regardless of the distribution's shape. - The KV cache makes each new token cheap to compute but uses memory that grows with context length. ## D5 · Frontier D2's loop of predict, pick, repeat is accurate, but it undersells what happens inside and hides real limits of the training objective. Planning inside. Interpretability work found a model picking a rhyme word before writing the line that leads to it, though the method gave clear insight on only about a quarter of prompts tried (Lindsey et al., 2025). Teacher forcing can fail. On a simple path-finding task, next-token training failed to learn the task at all, while predicting several tokens ahead helped (Bachmann and Nagarajan, 2024). Several tokens at once. Training to predict four tokens ahead improved coding results at 13B and allowed up to three times faster inference (Gloeckle et al., 2024). Not left to right. An 8B diffusion language model trained from scratch was competitive with similar autoregressive models, so left-to-right order is not the only option (Nie et al., 2025). ## Examples as of 2026-10-30 - Every chat model: GPT, Claude, Gemini, Llama and others - Sampling settings in model APIs: temperature and top-p ## Sources - Attention Is All You Need: https://arxiv.org/abs/1706.03762 - On the Biology of a Large Language Model: https://transformer-circuits.pub/2025/attribution-graphs/biology.html - Better & Faster Large Language Models via Multi-token Prediction: https://arxiv.org/abs/2404.19737 - Fast Inference from Transformers via Speculative Decoding: https://arxiv.org/abs/2211.17192 - The pitfalls of next-token prediction (Bachmann and Nagarajan): https://arxiv.org/abs/2403.06963 - Large Language Diffusion Models (Nie et al.): https://arxiv.org/abs/2502.09992 --- Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.