# Layer 5 · Foundation Models

URL: https://justuju.org/layer/foundation-models/

Large neural networks trained on broad data that can be adapted to many tasks.

Gives the layer above: A general capability, predicting and generating text and other media, packaged as model weights.
Needs from the layer below: Training data from Data and large clusters from Cloud & Infrastructure.

## Concepts

- [Tokens & tokenization](https://justuju.org/concept/tokenization/): Text is chopped into tokens (common words or pieces of words), and each token becomes a number the model can work with.
- [Transformers & attention](https://justuju.org/concept/transformer-attention/): The architecture behind modern language models: stacked layers in which every token can look back at earlier tokens and pull in what's relevant.
- [How a model predicts the next token](https://justuju.org/concept/next-token-prediction/): A language model reads the text so far, gives every possible next token a probability, picks one, and repeats.
- [Pretraining](https://justuju.org/concept/pretraining/): Training a model from scratch on huge amounts of text by asking it, over and over, to predict the next token and correcting its mistakes.
- [Post-training: from predictor to assistant](https://justuju.org/concept/post-training/): Fine-tuning and reinforcement learning that turn a raw text predictor into a helpful, safer assistant.
- [Open vs closed weights](https://justuju.org/concept/open-weights/): Some models' weights can be downloaded and run anywhere; others can only be reached through the provider's service.
- [Reasoning models](https://justuju.org/concept/reasoning-models/): Models trained to work through a problem step by step before answering, trading extra time and tokens at answer time (test-time compute) for better results on hard tasks.
- [Specialised and fine-tuned models](https://justuju.org/concept/specialised-models/): Models adapted for one job, such as coding or a small focused task, by fine-tuning, distillation or community modification of open weights.

---
Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.