Layer 2 · Compute Hardware

The chips that do the maths: GPUs and other accelerators, their fast memory, and the links between them.

In simple words: The chips that do the maths for AI, and the fast memory that feeds them.

Draft, awaiting expert review, reviewed 2026-10-02

Like: A kitchen: chips are the cooks, memory is the counter space, and interconnect is how fast cooks can pass dishes to each other.
Where it breaks: Cooks work independently. Thousands of accelerators must act as one machine in lockstep, so the network between them matters as much as the chips.

What it gives the layer above

Raw arithmetic throughput and memory, exposed to software through platforms such as CUDA or ROCm.

What it needs from the layer below

Power, cooling and fabricated silicon from Physical & Energy.

Concepts

• GPUs & AI accelerators
  Chips built to do enormous numbers of simple multiplications in parallel, the core operation of neural networks.
• Number formats (FP16, BF16, FP8, FP4)
  How many bits each number gets inside the chip. Fewer bits mean more maths per second and more numbers per trip from memory, at some cost in precision.
• Memory & bandwidth (HBM)
  How fast data reaches the chip often limits AI speed more than the chip's raw arithmetic.
• Chip-to-chip interconnect
  The links that let several accelerators share work. Fast dedicated links let a group of chips act like one big chip; slower general-purpose links make them wait.
• Local AI hardware
  Running models on your own machine: consumer graphics cards, multi-GPU home rigs, or desktop machines with large unified memory. Memory size decides what fits; memory speed decides how fast it runs.

Walkthrough

1. A home user wants a 70B model. Someone wants to run a 70-billion-parameter open model at home. The work is billions of multiplications per token, which is what a GPU's thousands of cores are for.
2. Fewer bits per number. At 16 bits the weights need about 140 GB. Stored at 4 bits they need about 35 GB, and the chip moves and multiplies far more numbers per second.
3. Fit first, then speed. 35 GB won't fit in one consumer card's memory. Once it fits somewhere, every token re-reads those weights, so memory bandwidth sets the tokens per second.
4. Two cards, one link. Two cards split the layers between them and pass a small result across PCIe for each token. Splitting inside every layer would need a much faster link.
5. The same story at data-centre scale. Frontier models face the same limits, scaled up: low-precision formats, stacked HBM, and NVLink-style links that join many accelerators into the clusters of the layer above.

Examples

• GPUs, TPUs and custom accelerators
• High-bandwidth memory (HBM)
• Low-precision formats such as BF16, FP8 and FP4
• NVLink-style links and open standards such as UALink
• Home hardware: consumer RTX cards, DGX Spark, Apple silicon