JustujuAI, layer by layer.Open in the interactive map

Quantization

Storing a model's numbers with fewer bits so it's smaller and faster, at a small cost in accuracy.

Layer 6 · Inference & Optimization · Run time · reviewed 2026-10-30

Read first: Memory & bandwidth (HBM).

D1 · StoryFor young children and total beginners

Which is easier to write, £3.99 or £3.98765? What do you lose?

Imagine writing a price as £3.99 instead of £3.98765. You lose a tiny bit of detail, but it's much quicker to write and read.

Quantization does this to the billions of numbers inside an AI so it fits on smaller computers.

Like: Rounding prices to the nearest penny.

D2 · PictureFor curious beginners

Model weights are usually trained in 16-bit formats. Quantization maps them to 8, 4 or fewer bits, using a scale factor for each small block of weights.

Size roughly follows the bits: memory ≈ parameters × bits ÷ 8. Because generation is often limited by memory reads, smaller weights also mean faster generation.

Quality loss is usually small at 8 bits and grows at lower precision; smarter methods protect the most sensitive weights.

Like: Rounding prices to the nearest penny.
Where it breaks: Rounding prices loses the same tiny amount everywhere. In a model, a few outlier weights matter far more than the rest, so naive rounding can break a model while careful methods barely change it.

D3 · PlayFor motivated learners

Quantization stores each weight with fewer bits. With b bits a weight can only take one of a fixed set of values on a grid, so each one is rounded to the nearest grid point.

Memory follows the bits exactly: memory = parameters × bits ÷ 8 bytes. The rounding error does not: pick a bit-width below and compare the two.

Try the interactive version in the map.

Things to try

D4 · MechanismFor practitioners

Real quantization splits the weights into blocks and gives each block its own scale: the block's largest absolute weight divided by the largest grid value. Each weight is stored as a small integer q = round(w ÷ scale).

One unusually large weight, an outlier, sets the scale for its whole block. The grid stretches to reach it, so every ordinary weight in that block rounds badly.

Smaller blocks contain the damage, but every block stores a 16-bit scale, so the real cost per weight is bits + 16 ÷ block size.

Try the interactive version in the map.

Things to try

for block in split(weights, block_size):
    scale = max(abs(w) for w in block) / (2 ** (bits - 1) - 1)
    q = [round(w / scale) for w in block]         # small integers, stored in `bits` bits
    store(q, scale)                                 # plus one 16-bit scale per block

# at run time
w_approx = q * scale

Trade-offs

D5 · FrontierFor experts

"Fewer bits, slightly worse" hides that the loss is uneven: it depends on the model, the method, the task and the bit width.

Outliers. A few very large values in the activations break simple rounding; handling them separately in higher precision made 8-bit inference work for large models (LLM.int8(), 2022).

Where the cliff is. Across 35,000 experiments, 4 bits was almost always the best trade-off between size and accuracy (Dettmers and Zettlemoyer, 2022), yet a newer, more heavily trained model lost noticeably more at very low bit widths (2024 study).

Formats in hardware. Block-scaled formats share one scale per small block of numbers; their authors trained language models below 8 bits with minimal accuracy loss (Microscaling, 2023). Which formats will last is still settling.

Behaviour, not just accuracy. Benchmarks can miss changes in behaviour: researchers built a model that looks harmless in full precision but turns malicious once users quantize it (Egashira et al., 2024).

Examples as of 2026-10-30

Sources

  1. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022)
  2. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022)
  3. AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (2023)
  4. The case for 4-bit precision: k-bit Inference Scaling Laws (2022-12-19)
  5. An empirical study of LLaMA3 quantization: from LLMs to MLLMs (2024-04-22)
  6. Microscaling Data Formats for Deep Learning (2023-10-16)
  7. Exploiting LLM Quantization (2024-05-28)

How it connects

More in Layer 6 · Inference & Optimization