JustujuAI, layer by layer.Open in the interactive map

Pretraining

Training a model from scratch on huge amounts of text by asking it, over and over, to predict the next token and correcting its mistakes.

Layer 5 · Foundation Models · Build time · reviewed 2026-10-30

Read first: Tokens & tokenization, Transformers & attention.

D1 · StoryFor young children and total beginners

Try it: cover the next word in a sentence and guess it.

Imagine reading millions of books where the next word is always covered. You guess, peek, and learn a tiny bit from every wrong guess.

Do that trillions of times and you become very good at guessing. That's pretraining.

Guess, peek, learn, repeat

  1. Read: a sentence, next word covered
  2. Guess: the covered word
  3. Peek: was the guess right?
  4. Learn: a tiny bit from the miss

Like: Guess the covered word, then peek.

D2 · PictureFor curious beginners

The training objective is simple: for each position in a batch of text, predict the next token. The difference between the prediction and the real token is the loss.

Gradient descent nudges every weight slightly to reduce that loss, batch after batch, on large GPU clusters.

Loss tends to fall predictably as models, data and compute grow together (scaling laws). The result is a base model that continues text but doesn't yet follow instructions.

The pretraining loop

  1. Batch: of real text
  2. Predict: next tokens
  3. Compare: loss vs real
  4. Adjust: all weights slightly

Like: Guess the covered word, then peek.
Where it breaks: A reader understands the story while guessing. The model only adjusts numbers to make the right token more likely; abilities like arithmetic or translation emerge unevenly from that single objective.

D3 · PlayFor motivated learners

Pretraining is one loop repeated many times: predict the next piece of text, compare with what really came next, and nudge the model's numbers so the right answer gets a little more likely.

Here is that loop at toy size. The model has one score for every pair of characters, so it can only look one character back. Press Train and watch the loss fall and the guesses sharpen.

Try the interactive version in the map.

Things to try

D4 · MechanismFor practitioners

The model is a table of scores W[a][b]. Softmax turns row a into probabilities p(b|a) for the character after a. The loss is the average of −log p of the character that really came next, over every pair in the text.

The gradient for each score is simple: (n_a × p(b|a) − count(a→b)) ÷ N, where n_a is how often a appears and N is the number of pairs. Gradient descent subtracts the learning rate η times that gradient from every score, every step.

Change η and watch the curve. Too small and it crawls. Too large and each step overshoots, so the loss bounces or climbs.

Try the interactive version in the map.

Things to try

W = small_random(V, V)              # W[a][b]: score for b following a
repeat steps times:
    loss, grad = 0, zeros(V, V)
    for a, b in pairs(text):        # every character and the one after it
        p = softmax(W[a])           # predict
        loss -= log(p[b])           # compare
        grad[a] += p                # push every guess down a little...
        grad[a][b] -= 1             # ...and the right answer up
    W -= lr * grad / N              # nudge (N = number of pairs)

Trade-offs

D5 · FrontierFor experts

D2's scaling laws say loss falls predictably. What to scale, and whether abilities follow loss, are still argued.

Compute-optimal or not. Compute-optimal training scales parameters and tokens equally (Hoffmann et al., 2022), but once serving demand is counted, smaller models trained for longer come out cheaper (Sardana et al., 2023).

Emergence. Abilities that seem to appear suddenly with scale may be an artefact of all-or-nothing metrics; smoother metrics show steadier gains (Schaeffer et al., 2023).

Reusing data. With limited data, up to about four passes over the same text was nearly as good as fresh text, but the value of more passes falls towards zero (Muennighoff et al., 2023).

Cost. The amortised cost of the largest training runs has grown about 2.4 times a year since 2016, so who can afford the next step is a strategic question (Cottier et al., 2024).

Examples as of 2026-10-30

Sources

  1. Scaling Laws for Neural Language Models (2020)
  2. Training Compute-Optimal Large Language Models (Chinchilla) (2022)
  3. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (Sardana et al.) (2023-12-31)
  4. Are Emergent Abilities of Large Language Models a Mirage? (Schaeffer et al.) (2023-04-28)
  5. Scaling Data-Constrained Language Models (Muennighoff et al.) (2023-05-25)
  6. The rising costs of training frontier AI models (Cottier et al.) (2024-05-31)

How it connects

More in Layer 5 · Foundation Models