JustujuAI, layer by layer.Open in the interactive map

Local AI hardware

Running models on your own machine: consumer graphics cards, multi-GPU home rigs, or desktop machines with large unified memory. Memory size decides what fits; memory speed decides how fast it runs.

Layer 2 · Compute Hardware · Run time · reviewed 2026-10-30

Read first: GPUs & AI accelerators, Memory & bandwidth (HBM), Number formats (FP16, BF16, FP8, FP4).

D1 · StoryFor young children and total beginners

Guess first: what stops a home computer running a huge model, speed or space?

You can run some AI models on a computer at home instead of in a giant data centre.

The main question is whether the model fits in the computer's fast memory. If it fits, it runs. If not, it crawls or won't start.

Like: Whether a sofa fits through your front door, before you worry how fast you can carry it.

D2 · PictureFor curious beginners

Weight memory is roughly parameters × bits ÷ 8, plus extra room for the context (the KV cache). A 70-billion-parameter model at 4 bits needs about 35 GB for weights alone.

Home setups take three common shapes. A consumer graphics card has very fast memory, but not much of it. Several cards in one machine pool their memory, usually splitting the model by layers over PCIe. Unified-memory machines let the processor and graphics share one large pool, so bigger models fit, but that memory is usually slower than a graphics card's.

Once a model fits, single-user speed follows memory bandwidth. If it doesn't fit, tools can spill layers into ordinary system memory, which works but is much slower.

Like: A sofa and a front door: first it has to fit, then you care how fast you can move it.
Where it breaks: A sofa either fits or it doesn't. A model can be shrunk with quantization or partly kept in slower system memory, so 'fits' is a sliding trade between size, speed and quality.

D5 · FrontierFor experts

D2's rule that speed follows memory bandwidth once a model fits breaks at the edges.

Mixture of experts. Only a few experts run per token, so idle experts can sit in slower memory. One study ran a large mixture-of-experts model on desktop hardware this way (Eliseev and Mazur, 2023).

Reading the prompt. Processing the prompt is compute-heavy while writing the answer is memory-heavy, so a machine that generates quickly can still be slow to read a long document (Patel et al., 2023).

Context memory. The KV cache grows with context. Quantising it to 2 bits cut peak memory 2.6 times in one study, but how far that holds across models is open (Liu et al., 2024).

Low-bit quality. A popular open model still lost noticeable quality at very low bit widths across ten quantisation methods, so squeezing a model to fit has a real price (Huang et al., 2024).

Examples as of 2026-10-30

Sources

  1. llama.cpp: LLM inference in C/C++ (GitHub) (2026)
  2. MLX: an array framework for Apple silicon (GitHub) (2026)
  3. NVIDIA DGX Spark product page (2026)
  4. Apple Mac Studio technical specifications (M5 Max, M5 Ultra) (2026)
  5. NVIDIA DGX Station product page (GB300) (2026)
  6. AMD Ryzen AI Halo product page (2026)
  7. AMD Ryzen AI Max+ 395 specifications (2026)
  8. NVIDIA GeForce RTX 5090 product page (2026)
  9. NVIDIA RTX PRO 6000 Blackwell Workstation Edition product page (2026)
  10. Fast Inference of Mixture-of-Experts Language Models with Offloading (Eliseev and Mazur) (2023-12-28)
  11. Splitwise: Efficient generative LLM inference using phase splitting (2023)
  12. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (Liu et al.) (2024-02-05)
  13. An empirical study of LLaMA3 quantization: from LLMs to MLLMs (Huang et al.) (2024-04-22)

How it connects

More in Layer 2 · Compute Hardware