Layer 4 · Data
The text, code, images and feedback that models learn from, and how it's collected, cleaned and licensed.
In simple words: Everything the AI learns from: web pages, books, code, pictures, and people saying which answer is better.
Like: Ingredients in a recipe: the best chef can't fix spoiled ingredients.
Where it breaks: Ingredients get used up; data doesn't, but reusing the same data again and again gives diminishing returns, and this 'ingredient' can carry hidden bias, private details or copyright obligations.
What it gives the layer above
Training corpora and feedback data that shape what a model can do.
What it needs from the layer below
Storage and compute from Cloud & Infrastructure to collect, filter and deduplicate at scale.
Concepts
- Pretraining data
The huge collection of text and other media a model first learns from, and how it is filtered. - Data curation and mixing
Removing copies and junk from raw data, then choosing how much of each kind of data a model trains on. - Synthetic data
Training data generated by models or programs rather than collected from people. - Human feedback and labelling
Example answers and side-by-side ratings written by people, used to teach a model to follow instructions and to prefer better answers. - Data rights and provenance
Tracking where training data came from, what licence it carries and whether its owners opted out, and the legal questions still open.
Walkthrough
- A recipe page is crawled. A blog post on boiling eggs is scooped up in a web crawl, along with thousands of copies of it on scraper sites.
- Is it allowed?. Before it's used, someone should check the blog's licence and whether its robots.txt opted out of AI crawlers. If it did, the page is dropped.
- Copies out, quality checked, mix set. Curation keeps one copy, scores the page as useful, and decides how much cooking text sits beside code, maths and books.
- Practice questions are generated. A model writes new questions and answers about cooking times; curation screens out the ones that are wrong before they reach post-training.
- People pick the better answer. A rater sees two answers to “How long do I boil an egg?” and picks the clearer one. That choice teaches post-training what people prefer.
Examples
- Web crawls such as Common Crawl
- Curated open datasets such as FineWeb
- Synthetic data from strong models
- Human labelling and preference data
- Data licensing deals and opt-outs