# Local AI hardware

URL: https://justuju.org/concept/local-ai-hardware/
Layer 2 · Compute Hardware

Running models on your own machine: consumer graphics cards, multi-GPU home rigs, or desktop machines with large unified memory. Memory size decides what fits; memory speed decides how fast it runs.

## D1 · Story

You can run some AI models on a computer at home instead of in a giant data centre.
The main question is whether the model fits in the computer's fast memory. If it fits, it runs. If not, it crawls or won't start.
Like: Whether a sofa fits through your front door, before you worry how fast you can carry it.

## D2 · Picture

Weight memory is roughly parameters × bits ÷ 8, plus extra room for the context (the KV cache). A 70-billion-parameter model at 4 bits needs about 35 GB for weights alone.
Home setups take three common shapes. A consumer graphics card has very fast memory, but not much of it. Several cards in one machine pool their memory, usually splitting the model by layers over PCIe. Unified-memory machines let the processor and graphics share one large pool, so bigger models fit, but that memory is usually slower than a graphics card's.
Once a model fits, single-user speed follows memory bandwidth. If it doesn't fit, tools can spill layers into ordinary system memory, which works but is much slower.
Like: A sofa and a front door: first it has to fit, then you care how fast you can move it. Where it breaks: A sofa either fits or it doesn't. A model can be shrunk with quantization or partly kept in slower system memory, so 'fits' is a sliding trade between size, speed and quality.

## D5 · Frontier

D2's rule that speed follows memory bandwidth once a model fits breaks at the edges.
Mixture of experts. Only a few experts run per token, so idle experts can sit in slower memory. One study ran a large mixture-of-experts model on desktop hardware this way (Eliseev and Mazur, 2023).
Reading the prompt. Processing the prompt is compute-heavy while writing the answer is memory-heavy, so a machine that generates quickly can still be slow to read a long document (Patel et al., 2023).
Context memory. The KV cache grows with context. Quantising it to 2 bits cut peak memory 2.6 times in one study, but how far that holds across models is open (Liu et al., 2024).
Low-bit quality. A popular open model still lost noticeable quality at very low bit widths across ten quantisation methods, so squeezing a model to fit has a real price (Huang et al., 2024).

## Examples as of 2026-10-30

- Consumer graphics cards: NVIDIA GeForce RTX 5090 (32 GB)
- Workstation graphics cards: NVIDIA RTX PRO 6000 Blackwell (96 GB)
- Unified-memory mini PCs: AMD Ryzen AI Max+ 395 machines and AMD's Ryzen AI Halo (128 GB, or 192 GB with the Max+ PRO 495)
- NVIDIA DGX Spark (128 GB unified memory)
- Apple Mac Studio with M5 Max (up to 128 GB) or M5 Ultra (up to 512 GB unified memory)
- NVIDIA DGX Station with GB300 (748 GB: 252 GB on the GPU plus 496 GB on the CPU)

## Sources

- llama.cpp: LLM inference in C/C++ (GitHub): https://github.com/ggml-org/llama.cpp
- MLX: an array framework for Apple silicon (GitHub): https://github.com/ml-explore/mlx
- NVIDIA DGX Spark product page: https://www.nvidia.com/en-us/products/workstations/dgx-spark/
- Apple Mac Studio technical specifications (M5 Max, M5 Ultra): https://www.apple.com/mac-studio/specs/
- NVIDIA DGX Station product page (GB300): https://www.nvidia.com/en-us/products/workstations/dgx-station/
- AMD Ryzen AI Halo product page: https://www.amd.com/en/products/processors/desktops/ryzen/ryzen-ai-halo.html
- AMD Ryzen AI Max+ 395 specifications: https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html
- NVIDIA GeForce RTX 5090 product page: https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/
- NVIDIA RTX PRO 6000 Blackwell Workstation Edition product page: https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000/
- Fast Inference of Mixture-of-Experts Language Models with Offloading (Eliseev and Mazur): https://arxiv.org/abs/2312.17238
- Splitwise: Efficient generative LLM inference using phase splitting: https://arxiv.org/abs/2311.18677
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (Liu et al.): https://arxiv.org/abs/2402.02750
- An empirical study of LLaMA3 quantization: from LLMs to MLLMs (Huang et al.): https://arxiv.org/abs/2404.14047


---
Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.