Local AI hardware
Running models on your own machine: consumer graphics cards, multi-GPU home rigs, or desktop machines with large unified memory. Memory size decides what fits; memory speed decides how fast it runs.
Read first: GPUs & AI accelerators, Memory & bandwidth (HBM), Number formats (FP16, BF16, FP8, FP4).
D1 · StoryFor young children and total beginners
You can run some AI models on a computer at home instead of in a giant data centre.
The main question is whether the model fits in the computer's fast memory. If it fits, it runs. If not, it crawls or won't start.
Like: Whether a sofa fits through your front door, before you worry how fast you can carry it.
D2 · PictureFor curious beginners
Weight memory is roughly parameters × bits ÷ 8, plus extra room for the context (the KV cache). A 70-billion-parameter model at 4 bits needs about 35 GB for weights alone.
Home setups take three common shapes. A consumer graphics card has very fast memory, but not much of it. Several cards in one machine pool their memory, usually splitting the model by layers over PCIe. Unified-memory machines let the processor and graphics share one large pool, so bigger models fit, but that memory is usually slower than a graphics card's.
Once a model fits, single-user speed follows memory bandwidth. If it doesn't fit, tools can spill layers into ordinary system memory, which works but is much slower.
Like: A sofa and a front door: first it has to fit, then you care how fast you can move it.
Where it breaks: A sofa either fits or it doesn't. A model can be shrunk with quantization or partly kept in slower system memory, so 'fits' is a sliding trade between size, speed and quality.
D5 · FrontierFor experts
Rules of thumb break at the edges. Mixture-of-experts models read only part of their weights per token, so they can run fast from slower but larger memory as long as all the weights fit.
Long contexts add a KV cache that can rival the weights in size, and prompt processing is limited by arithmetic rather than bandwidth, so a machine that generates quickly may still be slow to read a long document. Driver, software and format support often matter as much as the hardware itself.