Pretraining
Training a model from scratch on huge amounts of text by asking it, over and over, to predict the next token and correcting its mistakes.
Read first: Tokens & tokenization, Transformers & attention.
D1 · StoryFor young children and total beginners
Imagine reading millions of books where the next word is always covered. You guess, peek, and learn a tiny bit from every wrong guess.
Do that trillions of times and you become very good at guessing. That's pretraining.
Like: Guess the covered word, then peek.
D2 · PictureFor curious beginners
The training objective is simple: for each position in a batch of text, predict the next token. The difference between the prediction and the real token is the loss.
Gradient descent nudges every weight slightly to reduce that loss, batch after batch, on large GPU clusters.
Loss tends to fall predictably as models, data and compute grow together (scaling laws). The result is a base model that continues text but doesn't yet follow instructions.
The pretraining loop
- Batch: of real text
- Predict: next tokens
- Compare: loss vs real
- Adjust: all weights slightly
Like: Guess the covered word, then peek.
Where it breaks: A reader understands the story while guessing. The model only adjusts numbers to make the right token more likely; abilities like arithmetic or translation emerge unevenly from that single objective.
D3 · PlayFor motivated learners
Pretraining is one loop repeated many times: predict the next piece of text, compare with what really came next, and nudge the model's numbers so the right answer gets a little more likely.
Here is that loop at toy size. The model has one score for every pair of characters, so it can only look one character back. Press Train and watch the loss fall and the guesses sharpen.
Try the interactive version in the map.
Things to try
- Choose q, then press Train a few times. How sure does the model become about u?
- Choose e. Why does its top guess never get close to 100%?
- Press Write 60 characters before training, then again after 10 clicks of Train. What changed?
D4 · MechanismFor practitioners
The model is a table of scores W[a][b]. Softmax turns row a into probabilities p(b|a) for the character after a. The loss is the average of −log p of the character that really came next, over every pair in the text.
The gradient for each score is simple: (n_a × p(b|a) − count(a→b)) ÷ N, where n_a is how often a appears and N is the number of pairs. Gradient descent subtracts the learning rate η times that gradient from every score, every step.
Change η and watch the curve. Too small and it crawls. Too large and each step overshoots, so the loss bounces or climbs.
Try the interactive version in the map.
Things to try
- Set η to about 2 and press Step ×100. Then Reset, set η back to 20 and do the same. Compare the loss.
- Set η to about 500 and step. What does the curve do, and does the “best possible” line get any closer?
- Choose e and train until the loss stops falling. Compare the Share and p columns. What has the model learned?
W = small_random(V, V) # W[a][b]: score for b following a
repeat steps times:
loss, grad = 0, zeros(V, V)
for a, b in pairs(text): # every character and the one after it
p = softmax(W[a]) # predict
loss -= log(p[b]) # compare
grad[a] += p # push every guess down a little...
grad[a][b] -= 1 # ...and the right answer up
W -= lr * grad / N # nudge (N = number of pairs)Trade-offs
- A higher learning rate learns faster until it overshoots; real training warms the rate up and then lowers it over time.
- This toy uses the whole text every step. Real pretraining uses random batches: each step is far cheaper, but noisier.
- Looking one character back caps how good the model can get. Longer context and bigger models lower that floor, at the cost of much more compute.
- Lower training loss is not the goal on its own: a model that memorises its training text can do worse on new text.
D5 · FrontierFor experts
Open questions: how much more high-quality data exists, the balance between model size and training tokens (compute-optimal scaling), whether some abilities emerge suddenly or only look sudden, and how much synthetic data can substitute for human text.