Human feedback and labelling
Example answers and side-by-side ratings written by people, used to teach a model to follow instructions and to prefer better answers.
Read first: Pretraining data.
D1 · StoryFor young children and total beginners
After reading the library, the AI still needs to learn how to help. People write example answers to show it.
They also look at two answers side by side and pick the better one. Those choices teach the AI what people prefer.
Like: A teacher marking homework and writing model answers in the margin.
D2 · PictureFor curious beginners
Instruction data is a set of prompts with good answers written by people, used for supervised fine-tuning.
Preference data shows a rater two or more model answers to the same prompt and records which is better. A reward model learns from these choices.
Who makes it: contracted annotators (often through specialist labelling firms, sometimes in lower-wage countries), paid experts such as programmers or doctors for harder topics, and users clicking thumbs up or down in apps. Increasingly, AI models do some of the rating too.
How one preference label is made
- Prompt: a real task
- Two answers: from the model
- Human rater: picks the better one
- Preference pair: chosen vs rejected
- Reward model: learns the pattern
Like: A teacher marking two essays and saying which one is better.
Where it breaks: A teacher can explain why. A rater often has seconds per pair, may disagree with other raters, and may prefer answers that sound confident or flattering, and the model learns those habits too.
D5 · FrontierFor experts
Raters disagree, and whose preferences count is a choice with cultural and political weight. People cannot reliably grade answers to problems harder than they can solve, so feedback on expert tasks is costly and scarce.
Pay, working conditions and exposure to harmful content for annotators are ongoing concerns, and replacing people with AI raters risks copying a model's own blind spots.