# Synthetic data

URL: https://justuju.org/concept/synthetic-data/
Layer 4 · Data

Training data generated by models or programs rather than collected from people.

## D1 · Story

Sometimes a teacher writes practice questions instead of finding them in books. Synthetic data is like that. It is practice material written by one AI to help train another AI.
1. Teacher AI: writes practice questions
2. Check: throw out the bad ones
3. Practice set: the good ones
4. Student AI: learns from them
Like: A teacher writing practice questions.

## D2 · Picture

Models generate examples, explanations, conversations or code, which are filtered and then used for training.
It's most useful where answers can be checked automatically, such as maths or code with tests.
Risks: copying the generator's mistakes, and narrowing variety if models train on too much of their own output.
1. Generator: a strong model
2. Candidates: many examples
3. Verify: tests, checks, graders
4. Training set: keep the good ones
5. Student: trained model
Like: A teacher writing practice questions for a student. Where it breaks: A teacher knows more than the student. Synthetic data is only as good as the generator and the filter, so errors can be passed on with confidence.

## D5 · Frontier

D2 says synthetic data works best where answers can be checked. Beyond that, how far it can stand in for human data is disputed.
Collapse or not. Training each generation only on model output makes the rare tails of the data vanish (Shumailov et al., 2023). Keeping the real data and adding synthetic data on top avoided collapse in later experiments (Gerstgrasser et al., 2024).
Imitation is shallow. Fine-tuning smaller models on a stronger model's outputs mostly copied its style, not its accuracy, on tasks outside the imitation data (Gudibande et al., 2023).
Checkers set the limit. A 1.3B coding model trained partly on synthetic textbooks and exercises scored 50.6% on a standard test, where code can be checked; open-ended writing has no such check (Gunasekar et al., 2023).
Terms of service. Major providers forbid using their outputs to develop competing models, which leaves distillation from closed models commercially and legally uncertain (OpenAI Terms of Use).

## Examples as of 2026-10-30

- Microsoft's Phi models, trained heavily on synthetic text
- Cosmopedia (Hugging Face)
- NVIDIA Nemotron-4 340B, released for generating synthetic data
- Distilling reasoning traces from DeepSeek-R1

## Sources

- Textbooks Are All You Need: https://arxiv.org/abs/2306.11644
- The Curse of Recursion: Training on Generated Data Makes Models Forget: https://arxiv.org/abs/2305.17493
- Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data (Gerstgrasser et al.): https://arxiv.org/abs/2404.01413
- The False Promise of Imitating Proprietary LLMs (Gudibande et al.): https://arxiv.org/abs/2305.15717
- OpenAI Terms of Use: https://www.openai.com/policies/terms-of-use/


---
Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.