# Data curation and mixing

URL: https://justuju.org/concept/data-curation/
Layer 4 · Data

Removing copies and junk from raw data, then choosing how much of each kind of data a model trains on.

## D1 · Story

The web is full of copies, spam and half-broken pages. Curation means sorting through it and keeping the useful parts.
Then people decide the recipe: how much stories, how much code, how much maths. The mix changes what the AI is good at.
1. Big pile: copies, spam, good pages
2. Sort: keep the useful parts
3. Choose the mix: stories, code, maths
4. Small test AI: did it get better?
Like: Sorting a huge pile of second-hand books before stocking a school library.

## D2 · Picture

Deduplication removes exact and near-identical copies, so no page is learned far more often than it deserves.
Quality filters drop junk using simple rules (length, symbols, repeated lines) and small classifier models trained to spot 'good' text, such as educational writing.
Mixing sets the proportions of each source: web, code, books, maths, other languages. Teams test candidate mixes on small models before committing a large training run, because the mix shapes what the final model is good at.
1. Raw pool: crawls, code, books
2. Deduplicate: exact and near copies
3. Filter: rules and classifiers
4. Mix: set proportions
5. Small test model: measure results
Like: Stocking a library: throw out duplicates and torn copies, then decide how many shelves go to science, stories and maths. Where it breaks: A librarian can read every book. Curation at this scale is automated, so filters make millions of judgement calls nobody checks, and a 'quality' classifier can quietly drop dialects, minority languages or whole topics.

## D5 · Frontier

D2's dedupe, filter and mix loop works, but each step encodes choices that have no ground truth.
Whose quality. A widely used quality filter rated school newspapers from wealthier, more educated, urban areas as higher quality, so 'good text' classifiers carry social bias (Gururangan et al., 2022).
Filters trained on taste. A controlled benchmark found filtering with a model was key to the best training sets, which moves the question to what the filter model was trained to like (Li et al., 2024).
Small proxies. Mixture weights tuned on a 280M-parameter model let an 8B model reach baseline accuracy in 2.6 times fewer steps; whether small-model findings hold at frontier scale is not established (Xie et al., 2023).
How hard to deduplicate. Over 1% of models' unprompted output was copied verbatim from training data, and deduplication cut that tenfold; how aggressive to be is a judgement call (Lee et al., 2021).

## Examples as of 2026-10-30

- DataComp-LM (DCLM) filtering benchmark
- FineWeb-Edu quality classifier
- NVIDIA NeMo Curator
- datatrove (Hugging Face)

## Sources

- Deduplicating Training Data Makes Language Models Better: https://arxiv.org/abs/2107.06499
- DataComp-LM: In search of the next generation of training sets for language models: https://arxiv.org/abs/2406.11794
- DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining: https://arxiv.org/abs/2305.10429
- Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection (Gururangan et al.): https://arxiv.org/abs/2201.10474


---
Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.