Synthetic data

Training data generated by models or programs rather than collected from people.

Layer 4 · Data · Build time · reviewed 2026-10-30

D1 · StoryFor young children and total beginners

Think first: could practice questions written by a machine teach another machine?

Sometimes a teacher writes practice questions instead of finding them in books. Synthetic data is like that. It is practice material written by one AI to help train another AI.

Practice written by one AI to train another

Teacher AI: writes practice questions
Check: throw out the bad ones
Practice set: the good ones
Student AI: learns from them

Like: A teacher writing practice questions.

D2 · PictureFor curious beginners

Models generate examples, explanations, conversations or code, which are filtered and then used for training.

It's most useful where answers can be checked automatically, such as maths or code with tests.

Risks: copying the generator's mistakes, and narrowing variety if models train on too much of their own output.

A synthetic data pipeline

Generator: a strong model
Candidates: many examples
Verify: tests, checks, graders
Training set: keep the good ones
Student: trained model

Like: A teacher writing practice questions for a student.
Where it breaks: A teacher knows more than the student. Synthetic data is only as good as the generator and the filter, so errors can be passed on with confidence.

D3 · PlayFor motivated learners

What happens if a model learns only from the output of the model before it? This is model collapse in miniature, using the simplest possible model: a bell curve with a mean and a spread.

Generation 0 is fitted to real data. Each later generation draws a small sample from the previous fit and fits a new curve to that sample. Rare values at the edges, the tails, are seldom drawn, so they get lost first.

Step through the generations and watch the spread shrink. Then mix some real data into each round and see what changes.

Try the interactive version in the map.

Things to try

Press Next generation a few times, then drag to generation 60. What happens to the spread and to the share beyond ±2?
Reset, set real data mixed in to 20%, and drag to generation 60. Does the spread still head for zero?
Try 50% real data. Is the result steadier than at 20%?

D5 · FrontierFor experts

D2 says synthetic data works best where answers can be checked. Beyond that, how far it can stand in for human data is disputed.

Collapse or not. Training each generation only on model output makes the rare tails of the data vanish (Shumailov et al., 2023). Keeping the real data and adding synthetic data on top avoided collapse in later experiments (Gerstgrasser et al., 2024).

Imitation is shallow. Fine-tuning smaller models on a stronger model's outputs mostly copied its style, not its accuracy, on tasks outside the imitation data (Gudibande et al., 2023).

Checkers set the limit. A 1.3B coding model trained partly on synthetic textbooks and exercises scored 50.6% on a standard test, where code can be checked; open-ended writing has no such check (Gunasekar et al., 2023).

Terms of service. Major providers forbid using their outputs to develop competing models, which leaves distillation from closed models commercially and legally uncertain (OpenAI Terms of Use).

Examples as of 2026-10-30

Microsoft's Phi models, trained heavily on synthetic text
Cosmopedia (Hugging Face)
NVIDIA Nemotron-4 340B, released for generating synthetic data
Distilling reasoning traces from DeepSeek-R1

Sources

Textbooks Are All You Need (2023)
The Curse of Recursion: Training on Generated Data Makes Models Forget (2023)
Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data (Gerstgrasser et al.) (2024-04-01)
The False Promise of Imitating Proprietary LLMs (Gudibande et al.) (2023-05-25)
OpenAI Terms of Use (2026-01-01)

How it connects

Synthetic data supplies examples Post-training: from predictor to assistant.
Data curation and mixing checks and filters Synthetic data.
Synthetic data partly replaces Human feedback and labelling.