Layer 4 · Data

The text, code, images and feedback that models learn from, and how it's collected, cleaned and licensed.

In simple words: Everything the AI learns from: web pages, books, code, pictures, and people saying which answer is better.

reviewed 2026-10-30

Like: Ingredients in a recipe: the best chef can't fix spoiled ingredients.
Where it breaks: Ingredients get used up; data doesn't, but reusing the same data again and again gives diminishing returns, and this 'ingredient' can carry hidden bias, private details or copyright obligations.

Key players

What it means for your business: Your own data is often your real advantage, and you inherit the risks in everyone else's, so ask suppliers where their training data came from and what licences cover it.

• Common Crawl: Non-profit, free archive of the web; the raw base of many training sets.
• Hugging Face: Hosts open datasets and builds cleaned web sets such as FineWeb.
• Scale AI: Supplies human-labelled data to AI labs; Meta bought 49% in 2025.
• Reddit: Licenses its posts to AI companies, such as Google, for training.
• Wikimedia Enterprise: Paid feed of Wikipedia content used by firms such as Amazon, Meta and Mistral AI.

As of 2026-10-15. Sources: Common Crawl (ongoing); The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (2024); Meta invests $14B in Scale AI, acquires 49% stake (The Decoder) (2025-06-11); An expanded partnership with Reddit (Google) (2024-02-22); Wikipedia 25th anniversary: Enterprise partners announcement (2026-01-15)

What it gives the layer above

Training corpora and feedback data that shape what a model can do.

What it needs from the layer below

Storage and compute from Cloud & Infrastructure to collect, filter and deduplicate at scale.

Concepts

• Pretraining data
  The huge collection of text and other media a model first learns from, and how it is filtered.
• Data curation and mixing
  Removing copies and junk from raw data, then choosing how much of each kind of data a model trains on.
• Synthetic data
  Training data generated by models or programs rather than collected from people.
• Human feedback and labelling
  Example answers and side-by-side ratings written by people, used to teach a model to follow instructions and to prefer better answers.
• Data rights and provenance
  Tracking where training data came from, what licence it carries and whether its owners opted out, and the legal questions still open.

Walkthrough

1. A recipe page is crawled. A blog post on boiling eggs is scooped up in a web crawl, along with thousands of copies of it on scraper sites.
2. Is it allowed?. Before it's used, someone should check the blog's licence and whether its robots.txt opted out of AI crawlers. If it did, the page is dropped.
3. Copies out, quality checked, mix set. Curation keeps one copy, scores the page as useful, and decides how much cooking text sits beside code, maths and books.
4. Practice questions are generated. A model writes new questions and answers about cooking times; curation screens out the ones that are wrong before they reach post-training.
5. People pick the better answer. A rater sees two answers to “How long do I boil an egg?” and picks the clearer one. That choice teaches post-training what people prefer.

Examples

• Web crawls such as Common Crawl
• Curated open datasets such as FineWeb
• Synthetic data from strong models
• Human labelling and preference data
• Data licensing deals and opt-outs