GPUs & AI accelerators
Chips built to do enormous numbers of simple multiplications in parallel, the core operation of neural networks.
D1 · StoryFor young children and total beginners
A normal computer brain (a CPU) is like one very clever person doing sums one after another. A GPU is like a stadium full of people each doing one small sum at the same time.
AI needs billions of small sums, so the stadium wins.
Like: One clever person versus a stadium full of helpers.
D2 · PictureFor curious beginners
Neural networks mostly multiply large grids of numbers (matrices). GPUs and other accelerators contain thousands of simple cores and specialised matrix units that do these multiplications in parallel.
They also use lower-precision number formats (16-, 8- or even 4-bit) to fit more maths per second and more numbers in memory.
Each chip maker's software platform lets machine-learning frameworks use the hardware without chip-level code.
Like: A CPU is a few expert chefs; a GPU is a huge team of line cooks all chopping at once.
Where it breaks: Line cooks work independently. GPU cores run the same instruction on different data in lockstep, so work that can't be split evenly wastes most of the team.
D5 · FrontierFor experts
D2 explains why accelerators beat CPUs at matrix maths. It says nothing about who supplies them, what replaces them, or how fast they lose value.
Vendor concentration. The leading GPU maker's filing shows concentration on both sides: two direct customers made up 22% and 14% of its fiscal 2026 revenue, and it lists cloud companies designing their own hardware as competitors (NVIDIA 10-K, 2026).
Custom silicon. One cloud provider reported its fourth-generation in-house accelerator 1.2 to 1.7 times faster than a contemporary GPU on similar-sized systems, using less power. Whether in-house chips can keep pace and match a rival software ecosystem is open (Jouppi et al., 2023).
Price deflation. GPU compute per dollar has doubled roughly every 2.5 years, which erodes the value of fleets bought today (Hobbhahn and Besiroglu, 2022).
Memory wall. Over 20 years peak compute grew about 3.0 times every two years against 1.6 times for DRAM bandwidth, so feeding the arithmetic, not doing it, increasingly sets speed (Gholami et al., 2024).
Examples as of 2026-10-13
- NVIDIA GPUs, programmed with CUDA
- AMD GPUs, programmed with ROCm
- Google TPUs
- PyTorch (a framework that runs on all of these)
Sources
- Making Deep Learning Go Brrrr From First Principles (Horace He) (2022)
- In-Datacenter Performance Analysis of a Tensor Processing Unit (2017-04-16)
- NVIDIA Form 10-K, fiscal year 2026 (2026)
- TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings (Jouppi et al.) (2023-04-04)
- Trends in GPU price-performance (Hobbhahn and Besiroglu, Epoch AI) (2022-06-27)
- AI and Memory Wall (Gholami et al.) (2024-03-21)
How it connects
- Fabs & the chip supply chain manufactures GPUs & AI accelerators.
- GPUs & AI accelerators networked into GPU clusters & orchestration.
- GPUs & AI accelerators limited by Memory & bandwidth (HBM).
- Number formats (FP16, BF16, FP8, FP4) packs more maths into GPUs & AI accelerators.
- GPUs & AI accelerators share work through Chip-to-chip interconnect.
- Local AI hardware uses consumer versions of GPUs & AI accelerators.
More in Layer 2 · Compute Hardware
- Number formats (FP16, BF16, FP8, FP4)
How many bits each number gets inside the chip. Fewer bits mean more maths per second and more numbers per trip from memory, at some cost in precision. - Memory & bandwidth (HBM)
How fast data reaches the chip often limits AI speed more than the chip's raw arithmetic. - Chip-to-chip interconnect
The links that let several accelerators share work. Fast dedicated links let a group of chips act like one big chip; slower general-purpose links make them wait. - Local AI hardware
Running models on your own machine: consumer graphics cards, multi-GPU home rigs, or desktop machines with large unified memory. Memory size decides what fits; memory speed decides how fast it runs.