Quantization
Storing a model's numbers with fewer bits so it's smaller and faster, at a small cost in accuracy.
Read first: Memory & bandwidth (HBM).
D1 · StoryFor young children and total beginners
Imagine writing a price as £3.99 instead of £3.98765. You lose a tiny bit of detail, but it's much quicker to write and read.
Quantization does this to the billions of numbers inside an AI so it fits on smaller computers.
Like: Rounding prices to the nearest penny.
D2 · PictureFor curious beginners
Model weights are usually trained in 16-bit formats. Quantization maps them to 8, 4 or fewer bits, using a scale factor for each small block of weights.
Size roughly follows the bits: memory ≈ parameters × bits ÷ 8. Because generation is often limited by memory reads, smaller weights also mean faster generation.
Quality loss is usually small at 8 bits and grows at lower precision; smarter methods protect the most sensitive weights.
Like: Rounding prices to the nearest penny.
Where it breaks: Rounding prices loses the same tiny amount everywhere. In a model, a few outlier weights matter far more than the rest, so naive rounding can break a model while careful methods barely change it.
D3 · PlayFor motivated learners
Quantization stores each weight with fewer bits. With b bits a weight can only take one of a fixed set of values on a grid, so each one is rounded to the nearest grid point.
Memory follows the bits exactly: memory = parameters × bits ÷ 8 bytes. The rounding error does not: pick a bit-width below and compare the two.
Try the interactive version in the map.
Things to try
- Step from 16 to 8 to 4 bits. How does the memory change each time?
- Now go from 4 to 3 to 2 bits. Watch the dots jump away from the outlines.
- At 2 bits, how many different values can a weight take? Which weights end up exactly zero?
D4 · MechanismFor practitioners
Real quantization splits the weights into blocks and gives each block its own scale: the block's largest absolute weight divided by the largest grid value. Each weight is stored as a small integer q = round(w ÷ scale).
One unusually large weight, an outlier, sets the scale for its whole block. The grid stretches to reach it, so every ordinary weight in that block rounds badly.
Smaller blocks contain the damage, but every block stores a 16-bit scale, so the real cost per weight is bits + 16 ÷ block size.
Try the interactive version in the map.
Things to try
- At 4 bits with one block of 16, switch the outlier off and on. What happens to the other weights?
- With the outlier on, shrink the block size step by step. Which blocks' errors fall?
- Compare blocks of 2 with blocks of 16 in stored bits per weight. Is the smallest block worth it?
for block in split(weights, block_size):
scale = max(abs(w) for w in block) / (2 ** (bits - 1) - 1)
q = [round(w / scale) for w in block] # small integers, stored in `bits` bits
store(q, scale) # plus one 16-bit scale per block
# at run time
w_approx = q * scaleTrade-offs
- Smaller blocks give lower error but spend more memory on scales.
- Lower bit-widths save memory and bandwidth but error grows fast below about 4 bits.
- Outliers can be kept apart in higher precision, at the cost of extra bookkeeping.
- Quantizing after training is cheap; training with quantization in mind keeps more accuracy but costs a training run.
D5 · FrontierFor experts
Frontier: 8- and 4-bit number formats supported natively by hardware, quantization-aware training, and how quantization affects long-context accuracy and safety behaviour.