Synthetic data
Training data generated by models or programs rather than collected from people.
D1 · StoryFor young children and total beginners
Sometimes a teacher writes practice questions instead of finding them in books. Synthetic data is like that. It is practice material written by one AI to help train another AI.
Like: A teacher writing practice questions.
D2 · PictureFor curious beginners
Models generate examples, explanations, conversations or code, which are filtered and then used for training.
It's most useful where answers can be checked automatically, such as maths or code with tests.
Risks: copying the generator's mistakes, and narrowing variety if models train on too much of their own output.
A synthetic data pipeline
- Generator: a strong model
- Candidates: many examples
- Verify: tests, checks, graders
- Training set: keep the good ones
- Student: trained model
Like: A teacher writing practice questions for a student.
Where it breaks: A teacher knows more than the student. Synthetic data is only as good as the generator and the filter, so errors can be passed on with confidence.
D3 · PlayFor motivated learners
What happens if a model learns only from the output of the model before it? This is model collapse in miniature, using the simplest possible model: a bell curve with a mean and a spread.
Generation 0 is fitted to real data. Each later generation draws a small sample from the previous fit and fits a new curve to that sample. Rare values at the edges, the tails, are seldom drawn, so they get lost first.
Step through the generations and watch the spread shrink. Then mix some real data into each round and see what changes.
Try the interactive version in the map.
Things to try
- Press Next generation a few times, then drag to generation 60. What happens to the spread and to the share beyond ±2?
- Reset, set real data mixed in to 20%, and drag to generation 60. Does the spread still head for zero?
- Try 50% real data. Is the result steadier than at 20%?
D5 · FrontierFor experts
Open problems: quality loss when models train recursively on their own outputs ('model collapse'), keeping diversity, and legal and terms-of-service questions about learning from other providers' models.