Memory & bandwidth (HBM)
How fast data reaches the chip often limits AI speed more than the chip's raw arithmetic.
D1 · StoryFor young children and total beginners
Imagine a super-fast cook who has to wait for ingredients to come through a narrow door. The cook's speed doesn't matter if the door is slow.
For AI, the ingredients are the model's numbers, and the door is memory bandwidth.
Like: A fast cook waiting at a narrow pantry door.
D2 · PictureFor curious beginners
Generating each token means reading the model's active weights from memory. If those weights take tens of gigabytes, every new token moves that much data. On a home machine, whether those weights fit in fast memory at all is the first question (see local AI hardware).
High-bandwidth memory (HBM) stacks memory chips right next to the processor so data moves far faster than with ordinary memory.
Rule of thumb for a single user: tokens per second is roughly memory bandwidth divided by the bytes of weights read per token.
Why generation is often memory-bound
- Weights: in memory
- Read: limited by bandwidth
- Compute: fast, often waiting
- One token: then repeat
Like: A narrow door between the pantry and the kitchen.
Where it breaks: A pantry door moves each ingredient once. A model's weights are re-read for every single token, so the same data crosses the 'door' over and over; batching many users shares each trip.
D3 · PlayFor motivated learners
To write each new token for one user, the chip has to read every weight of the model from memory once. So the speed limit is simple: tokens per second ≤ memory bandwidth ÷ bytes read per token.
Change the model size, the bytes per weight and the bandwidth. Then turn the compute up and down and watch what happens.
Try the interactive version in the map.
Things to try
- With the 8B model in 16-bit, double the bandwidth. What happens to the upper bound?
- Now move the compute slider all the way down and back up. Does the answer change?
- Switch to 4-bit weights. Why does the bound go up even though the chip is the same?
D4 · MechanismFor practitioners
A roofline chart asks one question: how much work does the chip do for each byte it reads? That ratio is the arithmetic intensity, in FLOPs per byte.
Below the ridge point (peak compute ÷ bandwidth) the chip is memory-bound and speed follows the bandwidth slope. Above it, the chip is compute-bound and speed hits the flat compute roof.
Generating for one user does about 2 FLOPs per weight but reads the whole weight, so intensity is tiny. Serving many users in a batch reuses each weight read for every user, which moves the point right.
Try the interactive version in the map.
Things to try
- Start at batch 1 and move the batch slider up. At which batch size does the dot reach the roof?
- Switch from 16-bit to 8-bit weights. Why does the ridge now arrive at half the batch size?
- Double the compute. Which batch sizes get faster, and which do not change at all?
flops_per_step = 2 * params * batch # multiply and add for every weight, per user
bytes_per_step = params * bytes_per_weight # weights read once and shared by the batch
intensity = flops_per_step / bytes_per_step # = 2 * batch / bytes_per_weight
ridge = peak_flops / bandwidth
speed = min(peak_flops, bandwidth * intensity)
total_tokens_per_s = speed / (2 * params)
per_user_tokens_per_s = total_tokens_per_s / batchTrade-offs
- Bigger batches use the chip better but each user waits for the whole batch's step.
- Lower-precision weights raise intensity and speed, at some cost in accuracy.
- The KV cache adds memory reads that grow with context, so real serving reaches the roof later than this chart shows.
- Faster memory helps memory-bound work; more compute only helps work that is already past the ridge.
D5 · FrontierFor experts
Where it's heading: faster HBM generations, designs with much more on-chip memory, and algorithmic tricks such as speculative decoding that get more tokens out of each memory read.