# Quantization URL: https://justuju.org/concept/quantization/ Layer 6 · Inference & Optimization Storing a model's numbers with fewer bits so it's smaller and faster, at a small cost in accuracy. ## D1 · Story Imagine writing a price as £3.99 instead of £3.98765. You lose a tiny bit of detail, but it's much quicker to write and read. Quantization does this to the billions of numbers inside an AI so it fits on smaller computers. Like: Rounding prices to the nearest penny. ## D2 · Picture Model weights are usually trained in 16-bit formats. Quantization maps them to 8, 4 or fewer bits, using a scale factor for each small block of weights. Size roughly follows the bits: memory ≈ parameters × bits ÷ 8. Because generation is often limited by memory reads, smaller weights also mean faster generation. Quality loss is usually small at 8 bits and grows at lower precision; smarter methods protect the most sensitive weights. Like: Rounding prices to the nearest penny. Where it breaks: Rounding prices loses the same tiny amount everywhere. In a model, a few outlier weights matter far more than the rest, so naive rounding can break a model while careful methods barely change it. ## D4 · Mechanism Real quantization splits the weights into blocks and gives each block its own scale: the block's largest absolute weight divided by the largest grid value. Each weight is stored as a small integer q = round(w ÷ scale). One unusually large weight, an outlier, sets the scale for its whole block. The grid stretches to reach it, so every ordinary weight in that block rounds badly. Smaller blocks contain the damage, but every block stores a 16-bit scale, so the real cost per weight is bits + 16 ÷ block size. - Smaller blocks give lower error but spend more memory on scales. - Lower bit-widths save memory and bandwidth but error grows fast below about 4 bits. - Outliers can be kept apart in higher precision, at the cost of extra bookkeeping. - Quantizing after training is cheap; training with quantization in mind keeps more accuracy but costs a training run. ## D5 · Frontier "Fewer bits, slightly worse" hides that the loss is uneven: it depends on the model, the method, the task and the bit width. Outliers. A few very large values in the activations break simple rounding; handling them separately in higher precision made 8-bit inference work for large models (LLM.int8(), 2022). Where the cliff is. Across 35,000 experiments, 4 bits was almost always the best trade-off between size and accuracy (Dettmers and Zettlemoyer, 2022), yet a newer, more heavily trained model lost noticeably more at very low bit widths (2024 study). Formats in hardware. Block-scaled formats share one scale per small block of numbers; their authors trained language models below 8 bits with minimal accuracy loss (Microscaling, 2023). Which formats will last is still settling. Behaviour, not just accuracy. Benchmarks can miss changes in behaviour: researchers built a model that looks harmless in full precision but turns malicious once users quantize it (Egashira et al., 2024). ## Examples as of 2026-10-30 - GGUF quantised files for llama.cpp - GPTQ and AWQ - bitsandbytes - Quantised models for MLX on Apple silicon ## Sources - LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale: https://arxiv.org/abs/2208.07339 - GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers: https://arxiv.org/abs/2210.17323 - AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration: https://arxiv.org/abs/2306.00978 - The case for 4-bit precision: k-bit Inference Scaling Laws: https://arxiv.org/abs/2212.09720 - An empirical study of LLaMA3 quantization: from LLMs to MLLMs: https://arxiv.org/abs/2404.14047 - Microscaling Data Formats for Deep Learning: https://arxiv.org/abs/2310.10537 - Exploiting LLM Quantization: https://arxiv.org/abs/2405.18137 --- Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.