Pretraining data
The huge collection of text and other media a model first learns from, and how it is filtered.
D1 · StoryFor young children and total beginners
Before an AI can talk, it reads an enormous amount: websites, books and code. It's like a child who has read a whole library.
People try to remove junk and repeated pages first, because the AI learns whatever it reads.
Like: A child who has read an entire library, including the junk mail.
D2 · PictureFor curious beginners
Pretraining corpora are assembled from sources such as web crawls, code repositories, books and papers, then filtered.
Typical steps: extract the text, remove duplicates, filter low-quality or harmful content, detect language, and remove personal data where possible.
Quality matters as much as quantity: better-curated data can rival much larger, noisier datasets.
From the web to training tokens
- Collect: crawls, code, books
- Extract: clean text
- Deduplicate: remove copies
- Filter: quality and safety
- Mix: and tokenize
Like: Like panning for gold: most of what you scoop up is gravel.
Where it breaks: Gold either is gold or isn't. With data, 'quality' is a judgement call, and filters can remove useful dialects or topics along with the junk.
D5 · FrontierFor experts
Open questions: the limited supply of high-quality human text, licensing and consent, measuring data quality, and benchmark questions leaking into training data (contamination).