# Pretraining

URL: https://justuju.org/concept/pretraining/
Layer 5 · Foundation Models

Training a model from scratch on huge amounts of text by asking it, over and over, to predict the next token and correcting its mistakes.

## D1 · Story

Imagine reading millions of books where the next word is always covered. You guess, peek, and learn a tiny bit from every wrong guess.
Do that trillions of times and you become very good at guessing. That's pretraining.
1. Read: a sentence, next word covered
2. Guess: the covered word
3. Peek: was the guess right?
4. Learn: a tiny bit from the miss
Like: Guess the covered word, then peek.

## D2 · Picture

The training objective is simple: for each position in a batch of text, predict the next token. The difference between the prediction and the real token is the loss.
Gradient descent nudges every weight slightly to reduce that loss, batch after batch, on large GPU clusters.
Loss tends to fall predictably as models, data and compute grow together (scaling laws). The result is a base model that continues text but doesn't yet follow instructions.
1. Batch: of real text
2. Predict: next tokens
3. Compare: loss vs real
4. Adjust: all weights slightly
Like: Guess the covered word, then peek. Where it breaks: A reader understands the story while guessing. The model only adjusts numbers to make the right token more likely; abilities like arithmetic or translation emerge unevenly from that single objective.

## D4 · Mechanism

The model is a table of scores W[a][b]. Softmax turns row a into probabilities p(b|a) for the character after a. The loss is the average of −log p of the character that really came next, over every pair in the text.
The gradient for each score is simple: (n_a × p(b|a) − count(a→b)) ÷ N, where n_a is how often a appears and N is the number of pairs. Gradient descent subtracts the learning rate η times that gradient from every score, every step.
Change η and watch the curve. Too small and it crawls. Too large and each step overshoots, so the loss bounces or climbs.
- A higher learning rate learns faster until it overshoots; real training warms the rate up and then lowers it over time.
- This toy uses the whole text every step. Real pretraining uses random batches: each step is far cheaper, but noisier.
- Looking one character back caps how good the model can get. Longer context and bigger models lower that floor, at the cost of much more compute.
- Lower training loss is not the goal on its own: a model that memorises its training text can do worse on new text.

## D5 · Frontier

D2's scaling laws say loss falls predictably. What to scale, and whether abilities follow loss, are still argued.
Compute-optimal or not. Compute-optimal training scales parameters and tokens equally (Hoffmann et al., 2022), but once serving demand is counted, smaller models trained for longer come out cheaper (Sardana et al., 2023).
Emergence. Abilities that seem to appear suddenly with scale may be an artefact of all-or-nothing metrics; smoother metrics show steadier gains (Schaeffer et al., 2023).
Reusing data. With limited data, up to about four passes over the same text was nearly as good as fresh text, but the value of more passes falls towards zero (Muennighoff et al., 2023).
Cost. The amortised cost of the largest training runs has grown about 2.4 times a year since 2016, so who can afford the next step is a strategic question (Cottier et al., 2024).

## Examples as of 2026-10-30

- Llama 3 (Meta)
- DeepSeek-V3
- Qwen3 (Alibaba)
- OLMo 2 (Ai2, with open data and code)

## Sources

- Scaling Laws for Neural Language Models: https://arxiv.org/abs/2001.08361
- Training Compute-Optimal Large Language Models (Chinchilla): https://arxiv.org/abs/2203.15556
- Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (Sardana et al.): https://arxiv.org/abs/2401.00448
- Are Emergent Abilities of Large Language Models a Mirage? (Schaeffer et al.): https://arxiv.org/abs/2304.15004
- Scaling Data-Constrained Language Models (Muennighoff et al.): https://arxiv.org/abs/2305.16264
- The rising costs of training frontier AI models (Cottier et al.): https://arxiv.org/abs/2405.21015


---
Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.