How AI works, layer by layer

Layer 9 · Applications & Interfaces

Where people meet AI: chat assistants, copilots inside existing tools, and specialised apps.

• Chat assistants: General-purpose chat apps where you type, talk or upload files, and an assistant answers, writes, searches and uses tools.
• Voice & multimodal interfaces: Talking to AI out loud and showing it images or video, through phones, cars, speakers and glasses.
• AI features inside products: AI built into ordinary apps as buttons and features, where the app's own server calls a model API behind the scenes.
• Copilots: AI built into the tools people already work in, such as code editors, documents and email, suggesting and increasingly acting while the person stays in charge.

Layer 8 · Agents

Systems that pursue a goal over many steps: plan, act with tools, check the result, repeat.

• The agent loop: Plan, act with a tool, observe the result, and repeat until the goal is met.
• Agent harnesses: The program wrapped around a model that turns it into an agent: it runs tools, asks permission, manages context and keeps memory.
• Coding agents: Agents that work inside a code repository: read files, make edits, run commands and tests, and iterate until the checks pass.
• Computer & browser use: Agents that operate a screen like a person: look at a screenshot, then click, type and scroll.
• Multi-agent systems: Several agents with different roles or contexts work together on one task.

Layer 7 · Orchestration & Tooling

The glue around a model call: prompts, retrieval, tools, frameworks, evaluation and guardrails.

• Retrieval-augmented generation (RAG): Look up relevant documents first, then give them to the model to answer from.
• Tool use & protocols (MCP): Letting a model call functions and services through structured requests, standardised by protocols such as MCP.
• Memory: How an AI application remembers beyond one call: short-term memory in the context window, long-term memory in stores it saves to and searches.
• Context engineering: Choosing what goes into the model's context window, in what order, and what to leave out, because the model sees nothing else.
• Guardrails & evals: Checks around model calls (input and output filters, permission gates, prompt-injection defences) plus automated tests that measure whether the system still behaves.

Layer 6 · Inference & Optimization

Running trained models quickly and cheaply for real users: serving, APIs, compression and caching.

• Model APIs & routers: Paying per token to send requests to a hosted model, either directly to a provider or through a router that picks among many providers.
• Batching & KV caching: Serving tricks that process many users at once and avoid recomputing work already done.
• Quantization: Storing a model's numbers with fewer bits so it's smaller and faster, at a small cost in accuracy.
• Local inference runtimes: Programs that run downloaded model files on your own computer instead of a remote service.
• Speculative decoding: A small, fast model guesses the next few tokens and the big model checks them all at once: fewer slow steps, with output identical under greedy decoding and the same distribution when sampling.

Layer 5 · Foundation Models

Large neural networks trained on broad data that can be adapted to many tasks.

• Tokens & tokenization: Text is chopped into tokens (common words or pieces of words), and each token becomes a number the model can work with.
• Transformers & attention: The architecture behind modern language models: stacked layers in which every token can look back at earlier tokens and pull in what's relevant.
• How a model predicts the next token: A language model reads the text so far, gives every possible next token a probability, picks one, and repeats.
• Pretraining: Training a model from scratch on huge amounts of text by asking it, over and over, to predict the next token and correcting its mistakes.
• Post-training: from predictor to assistant: Fine-tuning and reinforcement learning that turn a raw text predictor into a helpful, safer assistant.
• Open vs closed weights: Some models' weights can be downloaded and run anywhere; others can only be reached through the provider's service.
• Reasoning models: Models trained to work through a problem step by step before answering, trading extra time and tokens at answer time (test-time compute) for better results on hard tasks.
• Specialised and fine-tuned models: Models adapted for one job, such as coding or a small focused task, by fine-tuning, distillation or community modification of open weights.

Layer 4 · Data

The text, code, images and feedback that models learn from, and how it's collected, cleaned and licensed.

• Pretraining data: The huge collection of text and other media a model first learns from, and how it is filtered.
• Data curation and mixing: Removing copies and junk from raw data, then choosing how much of each kind of data a model trains on.
• Synthetic data: Training data generated by models or programs rather than collected from people.
• Human feedback and labelling: Example answers and side-by-side ratings written by people, used to teach a model to follow instructions and to prefer better answers.
• Data rights and provenance: Tracking where training data came from, what licence it carries and whether its owners opted out, and the legal questions still open.

Layer 3 · Cloud & Infrastructure

Turns racks of chips into rentable, schedulable clusters for training and serving models.

• Cluster networking: The very fast network that lets thousands of accelerators share results every training step, so the whole cluster behaves like one computer.
• GPU clusters & orchestration: Thousands of accelerators networked and scheduled to act as one machine for training or serving.
• GPU clouds & neoclouds: Companies that own huge GPU clusters and rent slices of them by the hour, from big general clouds to specialist GPU-only providers.
• Training vs inference infrastructure: Training needs one giant, tightly linked cluster for weeks; serving answers needs many smaller copies close to users, sized for speed and cost per request.
• Running locally vs in the cloud: Open-weight models can run on your own hardware at home, trading the cloud's scale and convenience for privacy, control and a fixed cost.

Layer 2 · Compute Hardware

The chips that do the maths: GPUs and other accelerators, their fast memory, and the links between them.

• GPUs & AI accelerators: Chips built to do enormous numbers of simple multiplications in parallel, the core operation of neural networks.
• Number formats (FP16, BF16, FP8, FP4): How many bits each number gets inside the chip. Fewer bits mean more maths per second and more numbers per trip from memory, at some cost in precision.
• Memory & bandwidth (HBM): How fast data reaches the chip often limits AI speed more than the chip's raw arithmetic.
• Chip-to-chip interconnect: The links that let several accelerators share work. Fast dedicated links let a group of chips act like one big chip; slower general-purpose links make them wait.
• Local AI hardware: Running models on your own machine: consumer graphics cards, multi-GPU home rigs, or desktop machines with large unified memory. Memory size decides what fits; memory speed decides how fast it runs.

Layer 1 · Physical & Energy

The power, cooling, buildings and chip factories that everything else physically runs on.

• Grid connection & energy supply: A big AI data centre needs a large, steady supply of electricity, and getting connected to the grid can take years, so power, not chips, often decides where and when one gets built.
• Power & cooling for AI data centres: AI data centres are limited by how much electricity they can get and how much heat they can remove.
• Liquid cooling: AI racks pack so much power into a small space that fans and cold air can no longer carry the heat away, so water-based cooling is piped right up to the chips.
• Fabs & the chip supply chain: Advanced AI chips depend on a handful of highly specialised factories, tools and materials.
• Energy per token: Each answer a model gives costs a little electricity, and how much depends on what you count: the chips, the rest of the server, idle machines and the building's cooling.

Across every layer

• Safety & Alignment: Making models and agents behave as intended and avoid harm, from training data to deployed apps.
• Evaluation: Measuring what systems can do and how reliably, at every layer: benchmarks, red-teaming, production monitoring.
• Cost & Economics: Energy, chips, training runs and per-token prices: cost shapes decisions at every layer.
• Governance & Regulation: Laws, standards, licences and export controls that constrain what can be built and deployed.
• Security: Protecting every layer from attack: supply chain, infrastructure, model files, prompt injection and agent permissions.