Local inference runtimes
Programs that run downloaded model files on your own computer instead of a remote service.
Read first: Open vs closed weights, Quantization.
D1 · StoryFor young children and total beginners
Some AI models can be downloaded as one big file, like a film you keep instead of stream.
A local runtime is the player for that file. It runs the model on your own computer, so your words never leave the house.
The catch: the file has to fit in your computer's memory.
Like: Downloading a film instead of streaming it.
D2 · PictureFor curious beginners
A local runtime loads a model's weights into your GPU or CPU memory and generates tokens on your machine.
Models are usually shared as single files that bundle the weights, the tokenizer and settings such as the chat template. GGUF is a common format, and each model is often published in several quantization levels.
Most runtimes also start a small local server that speaks the same request format as the big cloud APIs, so existing apps and coding tools can point at your machine instead.
Memory decides what you can run: the weights (parameters × bits ÷ 8) plus the KV cache for your context must fit. Memory bandwidth then sets how fast tokens come out.
Like: Downloading a film instead of streaming it.
Where it breaks: A downloaded film plays the same on any player. A model file can run noticeably differently depending on the quantization level, the runtime's settings and whether the chat template matches, and a big model may not fit at all.
D5 · FrontierFor experts
The simple "does it fit" rule hides a lot. Splitting a model across several GPUs, or across GPU and system memory, works but slows generation, and machines with large shared memory change the trade-off between capacity and speed.
Quality at a given bit width varies by model and by quantization method, and community testing is often the only guide. Mismatched chat templates or tokenizer settings can quietly make a local model worse than the same model in the cloud.
Examples as of 2026-10-13
- llama.cpp
- Ollama
- LM Studio