Layer 5 · Foundation Models

Large neural networks trained on broad data that can be adapted to many tasks.

In simple words: The AI itself: a huge pattern-guesser that has read so much it can try almost any task.

Draft, awaiting expert review, reviewed 2026-10-02

Like: A widely read apprentice: has absorbed a huge library and can attempt almost anything, but needs direction to be useful.
Where it breaks: An apprentice remembers what they read. A model stores statistical patterns, not a searchable library, so it can produce fluent text that is simply wrong.

What it gives the layer above

A general capability, predicting and generating text and other media, packaged as model weights.

What it needs from the layer below

Training data from Data and large clusters from Cloud & Infrastructure.

Concepts

• Tokens & tokenization
  Text is chopped into tokens (common words or pieces of words), and each token becomes a number the model can work with.
• Transformers & attention
  The architecture behind modern language models: stacked layers in which every token can look back at earlier tokens and pull in what's relevant.
• How a model predicts the next token
  A language model reads the text so far, gives every possible next token a probability, picks one, and repeats.
• Pretraining
  Training a model from scratch on huge amounts of text by asking it, over and over, to predict the next token and correcting its mistakes.
• Post-training: from predictor to assistant
  Fine-tuning and reinforcement learning that turn a raw text predictor into a helpful, safer assistant.
• Open vs closed weights
  Some models' weights can be downloaded and run anywhere; others can only be reached through the provider's service.
• Reasoning models
  Models trained to work through a problem step by step before answering, trading extra time and tokens at answer time (test-time compute) for better results on hard tasks.
• Specialised and fine-tuned models
  Models adapted for one job, such as coding or a small focused task, by fine-tuning, distillation or community modification of open weights.

Walkthrough

1. Text becomes tokens. “The cat sat on the” is snapped into tokens, and each token becomes a vector of numbers.
2. Attention mixes in context. Layer after layer, each token looks back at the earlier ones, so “the” knows a cat is sitting.
3. One token is predicted. The last layer scores every possible next token; “mat” gets the highest probability, and one token is picked.
4. Pretraining set the weights. All of that only works because pretraining tuned billions of weights by predicting the next token across trillions of examples.
5. Post-training makes it an assistant, then it's released. Fine-tuning and feedback turn the predictor into a helpful assistant, released either as open weights or behind an API.
6. Then it's sharpened for a job. Extra training can teach a model to reason step by step before it answers, and open weights let anyone fine-tune specialists, such as coding models.

Examples

• Transformer architecture
• Closed and open-weight models
• Mixture-of-experts
• Reasoning models
• Coding models