# Memory & bandwidth (HBM) URL: https://justuju.org/concept/memory-bandwidth/ Layer 2 · Compute Hardware How fast data reaches the chip often limits AI speed more than the chip's raw arithmetic. ## D1 · Story Imagine a super-fast cook who has to wait for ingredients to come through a narrow door. The cook's speed doesn't matter if the door is slow. For AI, the ingredients are the model's numbers, and the door is memory bandwidth. 1. The pantry: the model's numbers, kept in memory 2. The narrow door: how fast numbers come out 3. The fast cook: the chip, often waiting 4. One new word: then fetch them all again Like: A fast cook waiting at a narrow pantry door. ## D2 · Picture Generating each token means reading the model's active weights from memory. If those weights take tens of gigabytes, every new token moves that much data. On a home machine, whether those weights fit in fast memory at all is the first question (see local AI hardware). High-bandwidth memory (HBM) stacks memory chips right next to the processor so data moves far faster than with ordinary memory. Rule of thumb for a single user: tokens per second is roughly memory bandwidth divided by the bytes of weights read per token. 1. Weights: in memory 2. Read: limited by bandwidth 3. Compute: fast, often waiting 4. One token: then repeat Like: A narrow door between the pantry and the kitchen. Where it breaks: A pantry door moves each ingredient once. A model's weights are re-read for every single token, so the same data crosses the 'door' over and over; batching many users shares each trip. ## D4 · Mechanism A roofline chart asks one question: how much work does the chip do for each byte it reads? That ratio is the arithmetic intensity, in FLOPs per byte. Below the ridge point (peak compute ÷ bandwidth) the chip is memory-bound and speed follows the bandwidth slope. Above it, the chip is compute-bound and speed hits the flat compute roof. Generating for one user does about 2 FLOPs per weight but reads the whole weight, so intensity is tiny. Serving many users in a batch reuses each weight read for every user, which moves the point right. - Bigger batches use the chip better but each user waits for the whole batch's step. - Lower-precision weights raise intensity and speed, at some cost in accuracy. - The KV cache adds memory reads that grow with context, so real serving reaches the roof later than this chart shows. - Faster memory helps memory-bound work; more compute only helps work that is already past the ridge. ## D5 · Frontier D4's roofline makes single-user generation a pure bandwidth problem. Real serving adds a growing cache, tricks that change the arithmetic, and limited memory supply. The KV cache. At long contexts and large batches the cache of past keys and values can rival the weights. Sharing key-value heads across queries comes close to full quality at much lower memory traffic (Ainslie et al., 2023). Speculative decoding. A small draft model proposes tokens and the large model checks several per weight read, giving 2 to 3 times speed-ups with identical outputs; the gain depends on how often drafts are accepted (Leviathan et al., 2022). Memory supply. High-bandwidth memory comes from very few makers. One said in October 2025 that it had already settled next year's HBM supply with key customers, so memory can cap accelerator output (SK hynix, 2025). A widening gap. DRAM bandwidth grew about 1.6 times every two years against 3.0 times for peak compute, so the ridge point keeps moving right (Gholami et al., 2024). ## Examples as of 2026-10-30 - HBM3E and HBM4 stacks from SK hynix, Samsung and Micron - NVIDIA Blackwell GPUs - AMD Instinct MI350-series GPUs - Google TPUs ## Sources - Roofline: an insightful visual performance model (Williams, Waterman, Patterson): https://doi.org/10.1145/1498765.1498785 - LLM Inference Unveiled: Survey and Roofline Model Insights: https://arxiv.org/abs/2402.16363 - GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al.): https://arxiv.org/abs/2305.13245 - Fast Inference from Transformers via Speculative Decoding (Leviathan et al.): https://arxiv.org/abs/2211.17192 - SK hynix Announces 3Q25 Financial Results: https://technode.global/prnasia/sk-hynix-announces-3q25-financial-results/ - AI and Memory Wall (Gholami et al.): https://arxiv.org/abs/2403.14123 --- Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.