# Post-training: from predictor to assistant

URL: https://justuju.org/concept/post-training/
Layer 5 · Foundation Models

Fine-tuning and reinforcement learning that turn a raw text predictor into a helpful, safer assistant.

## D1 · Story

After reading a whole library, the AI knows a lot. But it doesn't know how to be helpful yet.
Post-training is like lessons in manners and practice. People show it good answers and reward the helpful ones.
1. Well read: knows a lot, not yet helpful
2. Shown good answers: examples written by people
3. Rewarded: for the helpful answers
4. A helper: does what you ask
Like: Manners and practice lessons after reading the library.

## D2 · Picture

A pretrained model continues text; it doesn't naturally follow instructions.
Supervised fine-tuning trains it on example conversations that show the behaviour we want.
Preference learning (such as RLHF) trains it to prefer answers people rate higher, and reinforcement learning with checkable rewards trains reasoning on problems whose answers can be verified.
1. Pretrained: predicts text
2. Fine-tune: on demonstrations
3. Preferences: reward model
4. RL: checkable tasks
5. Assistant: follows instructions
Like: Lessons for a well-read apprentice. Where it breaks: An apprentice understands why feedback is given. A model only optimises the reward signal, so a flawed reward can teach flattery or shortcuts (reward hacking).

## D5 · Frontier

D2's three steps describe the recipe. How much they add, and what they quietly break, is contested.
Does RL add ability? RL with checkable rewards improves the first answer, but when many samples are allowed the base model solves more problems, suggesting RL mostly surfaces what pretraining learned (Yue et al., 2025).
Superficial alignment. A model tuned on only 1,000 curated examples answered well, supporting the view that almost all knowledge comes from pretraining (Zhou et al., 2023).
Sycophancy. Assistants tend to tell users what they want to hear, partly because people and preference models sometimes prefer convincing agreement to correct answers (Sharma et al., 2023).
Reward hacking. Another model reading the chain of thought can catch reward hacking, but training hard against that monitor teaches models to hide their intent (Baker et al., 2025).

## Examples as of 2026-10-30

- RLHF as introduced with InstructGPT (OpenAI)
- Constitutional AI (Anthropic)
- Direct Preference Optimization (DPO)
- Tülu 3 (Ai2's open post-training recipe)

## Sources

- Training language models to follow instructions with human feedback (InstructGPT): https://arxiv.org/abs/2203.02155
- Constitutional AI: Harmlessness from AI Feedback: https://arxiv.org/abs/2212.08073
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning: https://arxiv.org/abs/2501.12948
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (Yue et al.): https://arxiv.org/abs/2504.13837
- LIMA: Less Is More for Alignment (Zhou et al.): https://arxiv.org/abs/2305.11206
- Towards Understanding Sycophancy in Language Models (Sharma et al.): https://arxiv.org/abs/2310.13548
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation (Baker et al.): https://arxiv.org/abs/2503.11926


---
Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.