JustujuAI, layer by layer.Open in the interactive map

Data curation and mixing

Removing copies and junk from raw data, then choosing how much of each kind of data a model trains on.

Layer 4 · Data · Build time · reviewed 2026-10-30

Read first: Pretraining data.

D1 · StoryFor young children and total beginners

Guess first: is more data always better, or cleaner data?

The web is full of copies, spam and half-broken pages. Curation means sorting through it and keeping the useful parts.

Then people decide the recipe: how much stories, how much code, how much maths. The mix changes what the AI is good at.

Sort the pile, choose the recipe, test it

  1. Big pile: copies, spam, good pages
  2. Sort: keep the useful parts
  3. Choose the mix: stories, code, maths
  4. Small test AI: did it get better?

Like: Sorting a huge pile of second-hand books before stocking a school library.

D2 · PictureFor curious beginners

Deduplication removes exact and near-identical copies, so no page is learned far more often than it deserves.

Quality filters drop junk using simple rules (length, symbols, repeated lines) and small classifier models trained to spot 'good' text, such as educational writing.

Mixing sets the proportions of each source: web, code, books, maths, other languages. Teams test candidate mixes on small models before committing a large training run, because the mix shapes what the final model is good at.

Curation is tested in a loop on small models

  1. Raw pool: crawls, code, books
  2. Deduplicate: exact and near copies
  3. Filter: rules and classifiers
  4. Mix: set proportions
  5. Small test model: measure results

Like: Stocking a library: throw out duplicates and torn copies, then decide how many shelves go to science, stories and maths.
Where it breaks: A librarian can read every book. Curation at this scale is automated, so filters make millions of judgement calls nobody checks, and a 'quality' classifier can quietly drop dialects, minority languages or whole topics.

D5 · FrontierFor experts

D2's dedupe, filter and mix loop works, but each step encodes choices that have no ground truth.

Whose quality. A widely used quality filter rated school newspapers from wealthier, more educated, urban areas as higher quality, so 'good text' classifiers carry social bias (Gururangan et al., 2022).

Filters trained on taste. A controlled benchmark found filtering with a model was key to the best training sets, which moves the question to what the filter model was trained to like (Li et al., 2024).

Small proxies. Mixture weights tuned on a 280M-parameter model let an 8B model reach baseline accuracy in 2.6 times fewer steps; whether small-model findings hold at frontier scale is not established (Xie et al., 2023).

How hard to deduplicate. Over 1% of models' unprompted output was copied verbatim from training data, and deduplication cut that tenfold; how aggressive to be is a judgement call (Lee et al., 2021).

Examples as of 2026-10-30

Sources

  1. Deduplicating Training Data Makes Language Models Better (2021)
  2. DataComp-LM: In search of the next generation of training sets for language models (2024)
  3. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining (2023)
  4. Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection (Gururangan et al.) (2022-01-25)

How it connects

More in Layer 4 · Data