Post-training: from predictor to assistant
Fine-tuning and reinforcement learning that turn a raw text predictor into a helpful, safer assistant.
Read first: How a model predicts the next token.
D1 · StoryFor young children and total beginners
After reading a whole library, the AI knows a lot. But it doesn't know how to be helpful yet.
Post-training is like lessons in manners and practice. People show it good answers and reward the helpful ones.
Like: Manners and practice lessons after reading the library.
D2 · PictureFor curious beginners
A pretrained model continues text; it doesn't naturally follow instructions.
Supervised fine-tuning trains it on example conversations that show the behaviour we want.
Preference learning (such as RLHF) trains it to prefer answers people rate higher, and reinforcement learning with checkable rewards trains reasoning on problems whose answers can be verified.
From pretrained model to assistant
- Pretrained: predicts text
- Fine-tune: on demonstrations
- Preferences: reward model
- RL: checkable tasks
- Assistant: follows instructions
Like: Lessons for a well-read apprentice.
Where it breaks: An apprentice understands why feedback is given. A model only optimises the reward signal, so a flawed reward can teach flattery or shortcuts (reward hacking).
D5 · FrontierFor experts
Open problems: reward hacking and sycophancy, supervising models on tasks too hard for people to grade, and how much capability comes from post-training versus pretraining.