Data curation and mixing
Removing copies and junk from raw data, then choosing how much of each kind of data a model trains on.
Read first: Pretraining data.
D1 · StoryFor young children and total beginners
The web is full of copies, spam and half-broken pages. Curation means sorting through it and keeping the useful parts.
Then people decide the recipe: how much stories, how much code, how much maths. The mix changes what the AI is good at.
Like: Sorting a huge pile of second-hand books before stocking a school library.
D2 · PictureFor curious beginners
Deduplication removes exact and near-identical copies, so no page is learned far more often than it deserves.
Quality filters drop junk using simple rules (length, symbols, repeated lines) and small classifier models trained to spot 'good' text, such as educational writing.
Mixing sets the proportions of each source: web, code, books, maths, other languages. Teams test candidate mixes on small models before committing a large training run, because the mix shapes what the final model is good at.
Curation is tested in a loop on small models
- Raw pool: crawls, code, books
- Deduplicate: exact and near copies
- Filter: rules and classifiers
- Mix: set proportions
- Small test model: measure results
Like: Stocking a library: throw out duplicates and torn copies, then decide how many shelves go to science, stories and maths.
Where it breaks: A librarian can read every book. Curation at this scale is automated, so filters make millions of judgement calls nobody checks, and a 'quality' classifier can quietly drop dialects, minority languages or whole topics.
D5 · FrontierFor experts
There is no agreed definition of data quality, so filters encode the taste of whoever trained them. Results found on small test models do not always hold at full scale.
Benchmark questions that leak into the pool (contamination) can make a model look better than it is, and checking for them is hard at web scale.