Layer 5 · Foundation Models
Large neural networks trained on broad data that can be adapted to many tasks.
In simple words: The AI itself: a huge pattern-guesser that has read so much it can try almost any task.
Like: A widely read apprentice: has absorbed a huge library and can attempt almost anything, but needs direction to be useful.
Where it breaks: An apprentice remembers what they read. A model stores statistical patterns, not a searchable library, so it can produce fluent text that is simply wrong.
Key players
What it means for your business: Few software companies should train their own model; design so you can swap providers, test open-weight models you could host yourself, and expect the leading model to change often.
- OpenAI: Makes the GPT models behind ChatGPT and sells them through an API.
- Anthropic: Makes the Claude models; top of the 2026 AI Index arena ranking.
- Google DeepMind: Makes the Gemini models, trained and run on Google's own TPUs.
- xAI: Makes the Grok models; close behind the leaders in 2026 rankings.
- DeepSeek: Chinese lab releasing strong open-weight models under the MIT licence.
- Alibaba (Qwen): Releases the Qwen family of open-weight models under Apache 2.0.
- Mistral AI: Paris-based lab releasing open-weight models under Apache 2.0.
What it gives the layer above
A general capability, predicting and generating text and other media, packaged as model weights.
What it needs from the layer below
Training data from Data and large clusters from Cloud & Infrastructure.
Concepts
- Tokens & tokenization
Text is chopped into tokens (common words or pieces of words), and each token becomes a number the model can work with. - Transformers & attention
The architecture behind modern language models: stacked layers in which every token can look back at earlier tokens and pull in what's relevant. - How a model predicts the next token
A language model reads the text so far, gives every possible next token a probability, picks one, and repeats. - Pretraining
Training a model from scratch on huge amounts of text by asking it, over and over, to predict the next token and correcting its mistakes. - Post-training: from predictor to assistant
Fine-tuning and reinforcement learning that turn a raw text predictor into a helpful, safer assistant. - Open vs closed weights
Some models' weights can be downloaded and run anywhere; others can only be reached through the provider's service. - Reasoning models
Models trained to work through a problem step by step before answering, trading extra time and tokens at answer time (test-time compute) for better results on hard tasks. - Specialised and fine-tuned models
Models adapted for one job, such as coding or a small focused task, by fine-tuning, distillation or community modification of open weights.
Walkthrough
- Text becomes tokens. “The cat sat on the” is snapped into tokens, and each token becomes a vector of numbers.
- Attention mixes in context. Layer after layer, each token looks back at the earlier ones, so “the” knows a cat is sitting.
- One token is predicted. The last layer scores every possible next token; “mat” gets the highest probability, and one token is picked.
- Pretraining set the weights. All of that only works because pretraining tuned billions of weights by predicting the next token across trillions of examples.
- Post-training makes it an assistant, then it's released. Fine-tuning and feedback turn the predictor into a helpful assistant, released either as open weights or behind an API.
- Then it's sharpened for a job. Extra training can teach a model to reason step by step before it answers, and open weights let anyone fine-tune specialists, such as coding models.
Examples
- Transformer architecture
- Closed and open-weight models
- Mixture-of-experts
- Reasoning models
- Coding models