JustujuAI, layer by layer.Open in the interactive map

Pretraining data

The huge collection of text and other media a model first learns from, and how it is filtered.

Layer 4 · Data · Build time · reviewed 2026-10-30

D1 · StoryFor young children and total beginners

Guess first: what happens if the model reads lots of junk?

Before an AI can talk, it reads an enormous amount: websites, books and code. It's like a child who has read a whole library.

People try to remove junk and repeated pages first, because the AI learns whatever it reads.

Cleaning the library before the AI reads it

  1. Collect: websites, books and code
  2. Remove copies: the same page twice
  3. Remove junk: spam and broken pages
  4. Reading pile: what the AI learns from

Like: A child who has read an entire library, including the junk mail.

D2 · PictureFor curious beginners

Pretraining corpora are assembled from sources such as web crawls, code repositories, books and papers, then filtered.

Typical steps: extract the text, remove duplicates, filter low-quality or harmful content, detect language, and remove personal data where possible.

Quality matters as much as quantity: better-curated data can rival much larger, noisier datasets.

From the web to training tokens

  1. Collect: crawls, code, books
  2. Extract: clean text
  3. Deduplicate: remove copies
  4. Filter: quality and safety
  5. Mix: and tokenize

Like: Like panning for gold: most of what you scoop up is gravel.
Where it breaks: Gold either is gold or isn't. With data, 'quality' is a judgement call, and filters can remove useful dialects or topics along with the junk.

D5 · FrontierFor experts

D2 says curation beats raw size. At frontier scale the questions are whether enough good text exists, who may use it, and whether benchmark scores are clean.

Running out. If trends continue, training sets will match the whole stock of public human text between 2026 and 2032 (Villalobos et al., 2024).

Shrinking access. Within a year, about 5% of all tokens in a common web corpus became fully restricted through crawler rules, and 45% through terms of service (Longpre et al., 2024).

Contamination. On fresh maths questions written to mirror a popular benchmark, some models scored up to 8% lower (Zhang et al., 2024). Yet a black-box test found little sign of widespread contamination in five popular models (Oren et al., 2023), so how big the problem is remains disputed.

Examples as of 2026-10-30

Sources

  1. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (2024)
  2. Common Crawl (ongoing)
  3. Will we run out of data? Limits of LLM scaling based on human-generated data (Villalobos et al.) (2024-06-04)
  4. Consent in Crisis: The Rapid Decline of the AI Data Commons (2024)
  5. A Careful Examination of Large Language Model Performance on Grade School Arithmetic (Zhang et al.) (2024-05-01)
  6. Proving Test Set Contamination in Black Box Language Models (Oren et al.) (2023-10-26)

How it connects

More in Layer 4 · Data