Memory & bandwidth (HBM)

D1 · StoryFor young children and total beginners

Picture a narrow kitchen door. Does a faster cook help if ingredients arrive slowly?

Imagine a super-fast cook who has to wait for ingredients to come through a narrow door. The cook's speed doesn't matter if the door is slow.

For AI, the ingredients are the model's numbers, and the door is memory bandwidth.

The cook can only work as fast as the door

1. The pantry: the model's numbers, kept in memory
2. The narrow door: how fast numbers come out
3. The fast cook: the chip, often waiting
4. One new word: then fetch them all again

Like: A fast cook waiting at a narrow pantry door.

D2 · PictureFor curious beginners

Generating each token means reading the model's active weights from memory. If those weights take tens of gigabytes, every new token moves that much data. On a home machine, whether those weights fit in fast memory at all is the first question (see local AI hardware).

High-bandwidth memory (HBM) stacks memory chips right next to the processor so data moves far faster than with ordinary memory.

Rule of thumb for a single user: tokens per second is roughly memory bandwidth divided by the bytes of weights read per token.

Why generation is often memory-bound

1. Weights: in memory
2. Read: limited by bandwidth
3. Compute: fast, often waiting
4. One token: then repeat

Like: A narrow door between the pantry and the kitchen.
Where it breaks: A pantry door moves each ingredient once. A model's weights are re-read for every single token, so the same data crosses the 'door' over and over; batching many users shares each trip.

D3 · PlayFor motivated learners

To write each new token for one user, the chip has to read every weight of the model from memory once. So the speed limit is simple: tokens per second ≤ memory bandwidth ÷ bytes read per token.

Change the model size, the bytes per weight and the bandwidth. Then turn the compute up and down and watch what happens.

Try the interactive version in the map.

Things to try

• With the 8B model in 16-bit, double the bandwidth. What happens to the upper bound?
• Now move the compute slider all the way down and back up. Does the answer change?
• Switch to 4-bit weights. Why does the bound go up even though the chip is the same?

D4 · MechanismFor practitioners

A roofline chart asks one question: how much work does the chip do for each byte it reads? That ratio is the arithmetic intensity, in FLOPs per byte.

Below the ridge point (peak compute ÷ bandwidth) the chip is memory-bound and speed follows the bandwidth slope. Above it, the chip is compute-bound and speed hits the flat compute roof.

Generating for one user does about 2 FLOPs per weight but reads the whole weight, so intensity is tiny. Serving many users in a batch reuses each weight read for every user, which moves the point right.

Try the interactive version in the map.

Things to try

• Start at batch 1 and move the batch slider up. At which batch size does the dot reach the roof?
• Switch from 16-bit to 8-bit weights. Why does the ridge now arrive at half the batch size?
• Double the compute. Which batch sizes get faster, and which do not change at all?

flops_per_step = 2 * params * batch          # multiply and add for every weight, per user
bytes_per_step = params * bytes_per_weight    # weights read once and shared by the batch
intensity = flops_per_step / bytes_per_step   # = 2 * batch / bytes_per_weight
ridge = peak_flops / bandwidth
speed = min(peak_flops, bandwidth * intensity)
total_tokens_per_s = speed / (2 * params)
per_user_tokens_per_s = total_tokens_per_s / batch

Trade-offs

• Bigger batches use the chip better but each user waits for the whole batch's step.
• Lower-precision weights raise intensity and speed, at some cost in accuracy.
• The KV cache adds memory reads that grow with context, so real serving reaches the roof later than this chart shows.
• Faster memory helps memory-bound work; more compute only helps work that is already past the ridge.

D5 · FrontierFor experts

D4's roofline makes single-user generation a pure bandwidth problem. Real serving adds a growing cache, tricks that change the arithmetic, and limited memory supply.

The KV cache. At long contexts and large batches the cache of past keys and values can rival the weights. Sharing key-value heads across queries comes close to full quality at much lower memory traffic (Ainslie et al., 2023).

Speculative decoding. A small draft model proposes tokens and the large model checks several per weight read, giving 2 to 3 times speed-ups with identical outputs; the gain depends on how often drafts are accepted (Leviathan et al., 2022).

Memory supply. High-bandwidth memory comes from very few makers. One said in October 2025 that it had already settled next year's HBM supply with key customers, so memory can cap accelerator output (SK hynix, 2025).

A widening gap. DRAM bandwidth grew about 1.6 times every two years against 3.0 times for peak compute, so the ridge point keeps moving right (Gholami et al., 2024).

Examples as of 2026-10-30

• HBM3E and HBM4 stacks from SK hynix, Samsung and Micron
• NVIDIA Blackwell GPUs
• AMD Instinct MI350-series GPUs
• Google TPUs

Sources

1. Roofline: an insightful visual performance model (Williams, Waterman, Patterson) (2009)
2. LLM Inference Unveiled: Survey and Roofline Model Insights (2024-02-26)
3. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al.) (2023-05-22)
4. Fast Inference from Transformers via Speculative Decoding (Leviathan et al.) (2022-11-30)
5. SK hynix Announces 3Q25 Financial Results (2025-10-29)
6. AI and Memory Wall (Gholami et al.) (2024-03-21)

How it connects

• GPUs & AI accelerators limited by Memory & bandwidth (HBM).
• Batching & KV caching shares reads of Memory & bandwidth (HBM).
• Local AI hardware sized and paced by Memory & bandwidth (HBM).

More in Layer 2 · Compute Hardware

• GPUs & AI accelerators
  Chips built to do enormous numbers of simple multiplications in parallel, the core operation of neural networks.
• Number formats (FP16, BF16, FP8, FP4)
  How many bits each number gets inside the chip. Fewer bits mean more maths per second and more numbers per trip from memory, at some cost in precision.
• Chip-to-chip interconnect
  The links that let several accelerators share work. Fast dedicated links let a group of chips act like one big chip; slower general-purpose links make them wait.
• Local AI hardware
  Running models on your own machine: consumer graphics cards, multi-GPU home rigs, or desktop machines with large unified memory. Memory size decides what fits; memory speed decides how fast it runs.