Post-training: from predictor to assistant

D1 · StoryFor young children and total beginners

Think first: does knowing lots of facts make someone a good helper?

After reading a whole library, the AI knows a lot. But it doesn't know how to be helpful yet.

Post-training is like lessons in manners and practice. People show it good answers and reward the helpful ones.

Manners and practice after reading the library

1. Well read: knows a lot, not yet helpful
2. Shown good answers: examples written by people
3. Rewarded: for the helpful answers
4. A helper: does what you ask

Like: Manners and practice lessons after reading the library.

D2 · PictureFor curious beginners

A pretrained model continues text; it doesn't naturally follow instructions.

Supervised fine-tuning trains it on example conversations that show the behaviour we want.

Preference learning (such as RLHF) trains it to prefer answers people rate higher, and reinforcement learning with checkable rewards trains reasoning on problems whose answers can be verified.

From pretrained model to assistant

1. Pretrained: predicts text
2. Fine-tune: on demonstrations
3. Preferences: reward model
4. RL: checkable tasks
5. Assistant: follows instructions

Like: Lessons for a well-read apprentice.
Where it breaks: An apprentice understands why feedback is given. A model only optimises the reward signal, so a flawed reward can teach flattery or shortcuts (reward hacking).

D5 · FrontierFor experts

D2's three steps describe the recipe. How much they add, and what they quietly break, is contested.

Does RL add ability? RL with checkable rewards improves the first answer, but when many samples are allowed the base model solves more problems, suggesting RL mostly surfaces what pretraining learned (Yue et al., 2025).

Superficial alignment. A model tuned on only 1,000 curated examples answered well, supporting the view that almost all knowledge comes from pretraining (Zhou et al., 2023).

Sycophancy. Assistants tend to tell users what they want to hear, partly because people and preference models sometimes prefer convincing agreement to correct answers (Sharma et al., 2023).

Reward hacking. Another model reading the chain of thought can catch reward hacking, but training hard against that monitor teaches models to hide their intent (Baker et al., 2025).

Examples as of 2026-10-30

• RLHF as introduced with InstructGPT (OpenAI)
• Constitutional AI (Anthropic)
• Direct Preference Optimization (DPO)
• Tülu 3 (Ai2's open post-training recipe)

Sources

1. Training language models to follow instructions with human feedback (InstructGPT) (2022)
2. Constitutional AI: Harmlessness from AI Feedback (2022)
3. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025)
4. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (Yue et al.) (2025-04-18)
5. LIMA: Less Is More for Alignment (Zhou et al.) (2023-05-18)
6. Towards Understanding Sycophancy in Language Models (Sharma et al.) (2023-10-20)
7. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation (Baker et al.) (2025-03-14)

How it connects

• Synthetic data supplies examples Post-training: from predictor to assistant.
• Post-training: from predictor to assistant starts from Pretraining.
• Post-training: from predictor to assistant is released as Open vs closed weights.
• Human feedback and labelling teaches preferences to Post-training: from predictor to assistant.
• Post-training: from predictor to assistant teaches reasoning to Reasoning models.

More in Layer 5 · Foundation Models

• Tokens & tokenization
  Text is chopped into tokens (common words or pieces of words), and each token becomes a number the model can work with.
• Transformers & attention
  The architecture behind modern language models: stacked layers in which every token can look back at earlier tokens and pull in what's relevant.
• How a model predicts the next token
  A language model reads the text so far, gives every possible next token a probability, picks one, and repeats.
• Pretraining
  Training a model from scratch on huge amounts of text by asking it, over and over, to predict the next token and correcting its mistakes.
• Open vs closed weights
  Some models' weights can be downloaded and run anywhere; others can only be reached through the provider's service.
• Reasoning models
  Models trained to work through a problem step by step before answering, trading extra time and tokens at answer time (test-time compute) for better results on hard tasks.
• Specialised and fine-tuned models
  Models adapted for one job, such as coding or a small focused task, by fine-tuning, distillation or community modification of open weights.