# Human feedback and labelling

URL: https://justuju.org/concept/human-feedback-data/
Layer 4 · Data

Example answers and side-by-side ratings written by people, used to teach a model to follow instructions and to prefer better answers.

## D1 · Story

After reading the library, the AI still needs to learn how to help. People write example answers to show it.
They also look at two answers side by side and pick the better one. Those choices teach the AI what people prefer.
1. A question: a real task
2. Two answers: from the AI
3. A person: picks the better one
4. The AI learns: what people prefer
Like: A teacher marking homework and writing model answers in the margin.

## D2 · Picture

Instruction data is a set of prompts with good answers written by people, used for supervised fine-tuning.
Preference data shows a rater two or more model answers to the same prompt and records which is better. A reward model learns from these choices.
Who makes it: contracted annotators (often through specialist labelling firms, sometimes in lower-wage countries), paid experts such as programmers or doctors for harder topics, and users clicking thumbs up or down in apps. Increasingly, AI models do some of the rating too.
1. Prompt: a real task
2. Two answers: from the model
3. Human rater: picks the better one
4. Preference pair: chosen vs rejected
5. Reward model: learns the pattern
Like: A teacher marking two essays and saying which one is better. Where it breaks: A teacher can explain why. A rater often has seconds per pair, may disagree with other raters, and may prefer answers that sound confident or flattering, and the model learns those habits too.

## D5 · Frontier

D2 treats feedback as people choosing the better answer. That breaks when people disagree, cannot judge, or are replaced by models.
Whose preferences. Model opinions differed from those of 60 US demographic groups by about as much as Democrats and Republicans differ on climate change (Santurkar et al., 2023).
Judging beyond your skill. Feedback is scarce exactly where tasks exceed raters' ability. Having people work with an imperfect model assistant beat both alone in early scalable oversight experiments (Bowman et al., 2022).
Over-optimising the proxy. A reward model only stands in for people. How true quality changes as a policy pushes against it follows regular, size-dependent patterns that differ between RL and best-of-n sampling (Gao et al., 2022).
AI raters. AI feedback matched human feedback on summarisation and dialogue tasks; the open question is whether it copies the rating model's own blind spots (Lee et al., 2023).

## Examples as of 2026-10-30

- Scale AI and Surge AI (labelling companies)
- Anthropic's HH-RLHF preference dataset
- Arena (formerly LMArena) human votes
- Ai2's Tülu 3 preference data

## Sources

- Deep Reinforcement Learning from Human Preferences: https://arxiv.org/abs/1706.03741
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback: https://arxiv.org/abs/2204.05862
- Training language models to follow instructions with human feedback (InstructGPT): https://arxiv.org/abs/2203.02155
- Whose Opinions Do Language Models Reflect? (Santurkar et al.): https://arxiv.org/abs/2303.17548
- Measuring Progress on Scalable Oversight for Large Language Models (Bowman et al.): https://arxiv.org/abs/2211.03540
- Scaling Laws for Reward Model Overoptimization (Gao, Schulman, Hilton): https://arxiv.org/abs/2210.10760
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (Lee et al.): https://arxiv.org/abs/2309.00267
- LMArena is now Arena (Arena blog): https://arena.ai/blog/lmarena-is-now-arena


---
Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.