Number formats (FP16, BF16, FP8, FP4)

How many bits each number gets inside the chip. Fewer bits mean more maths per second and more numbers per trip from memory, at some cost in precision.

Layer 2 · Compute Hardware · Build and run time · reviewed 2026-10-30

Read first: GPUs & AI accelerators.

D1 · StoryFor young children and total beginners

Count the digits: which is quicker to multiply, 3.14159 or 3.1?

Every number inside an AI is stored with a set number of tiny on-off switches called bits.

Fewer bits is like writing 3.1 instead of 3.14159. It is less exact, but the chip can carry and multiply far more of them at once.

Like: Writing 3.1 instead of 3.14159 so you can do more sums in the same time.

D2 · PictureFor curious beginners

A floating-point number splits its bits into a sign, an exponent (how big or small the number can be) and a mantissa (how many significant digits it keeps). FP32 uses 32 bits, FP16 and BF16 use 16, FP8 uses 8 and FP4 uses 4.

BF16 keeps the same exponent range as FP32 but fewer digits, which suits neural networks: they care more about range than precision. FP8 comes in two flavours that trade range for digits in the same way.

Halving the bits roughly doubles how many numbers fit in memory and move per second, and the chip's matrix units can do more multiplications per second in the narrower format. Very small formats such as FP4 share one scale factor across a small block of numbers so they don't run out of range.

Like: Rounding to fewer decimal places so more numbers fit on the page.
Where it breaks: Decimal rounding drops digits evenly. Floating-point formats choose how to split bits between range and precision, and FP4 also needs a shared scale per block, so the format design matters as much as the bit count.

D5 · FrontierFor experts

D2 says fewer bits buy speed and memory for a small loss in quality. How few bits, for which numbers, and at which stage of training is not settled.

4-bit training. One team trained a 12B model on 10 trillion tokens in a 4-bit format with loss close to an 8-bit baseline, but needed random Hadamard transforms, stochastic rounding and some layers kept in higher precision (NVIDIA, 2025).

Long training quantises worse. Damage from quantising after training grows the more data a model was trained on, so the trend to longer training runs cuts against low-bit serving (Kumar et al., 2024).

Outliers. A few systematic outlier features dominate model behaviour. One fix keeps those dimensions in 16-bit while over 99.9% of values run in 8-bit, which sits awkwardly with uniform low-bit hardware (Dettmers et al., 2022).

Silicon decides. Block-scaled formats that share one scale per small block have trained models below 8 bits, but speed-ups only arrive on chips with matching matrix units (Rouhani et al., 2023).

Examples as of 2026-10-30

BF16 and FP8 on NVIDIA Hopper GPUs
NVFP4 on NVIDIA Blackwell GPUs
OCP Microscaling (MX) formats: MXFP8, MXFP6 and MXFP4
FP6 and FP4 support on AMD Instinct MI350-series GPUs

Sources

Mixed Precision Training (Micikevicius et al.) (2017)
BFloat16: The secret to high performance on Cloud TPUs (Google Cloud) (2019-08-23)
FP8 Formats for Deep Learning (Micikevicius et al.) (2022)
Microscaling Data Formats for Deep Learning (Rouhani et al.) (2023)
Pretraining Large Language Models with NVFP4 (NVIDIA) (2025-09-29)
Scaling Laws for Precision (Kumar et al.) (2024-11-07)
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (Dettmers et al.) (2022-08-15)

How it connects

Number formats (FP16, BF16, FP8, FP4) packs more maths into GPUs & AI accelerators.