# Local inference runtimes

URL: https://justuju.org/concept/local-inference/
Layer 6 · Inference & Optimization

Programs that run downloaded model files on your own computer instead of a remote service.

## D1 · Story

Some AI models can be downloaded as one big file, like a film you keep instead of stream.
A local runtime is the player for that file. It runs the model on your own computer, so your words never leave the house.
The catch: the file has to fit in your computer's memory.
Like: Downloading a film instead of streaming it.

## D2 · Picture

A local runtime loads a model's weights into your GPU or CPU memory and generates tokens on your machine.
Models are usually shared as single files that bundle the weights, the tokenizer and settings such as the chat template. GGUF is a common format, and each model is often published in several quantization levels.
Most runtimes also start a small local server that speaks the same request format as the big cloud APIs, so existing apps and coding tools can point at your machine instead.
Memory decides what you can run: the weights (parameters × bits ÷ 8) plus the KV cache for your context must fit. Memory bandwidth then sets how fast tokens come out.
Like: Downloading a film instead of streaming it. Where it breaks: A downloaded film plays the same on any player. A model file can run noticeably differently depending on the quantization level, the runtime's settings and whether the chat template matches, and a big model may not fit at all.

## D5 · Frontier

"Does it fit in memory?" is only the first question. Speed, quality and trust depend on details the rule of thumb hides.
Splitting a model. Runtimes can put some layers on the CPU when the GPU is full, so a model too big for the GPU still runs, but tokens then come out at the pace of the slower memory (llama.cpp).
Quality per bit varies. The same bit width costs some models more than others; one study found a newer model degraded noticeably at low bit widths (2024 study). Community testing is often the only guide.
Templates and settings. A wrong chat template or missing control tokens can make a model "drastically worse" with no error message (chat template docs), so a local model can trail the same model in the cloud.
Trusting downloaded files. Pickle-based files can run code when loaded (pickle scanning docs), and a model can be crafted to misbehave only after quantization (Egashira et al., 2024). Where a file came from matters.

## Examples as of 2026-10-13

- llama.cpp
- Ollama
- LM Studio

## Sources

- GGUF file format specification (ggml): https://github.com/ggml-org/ggml/blob/master/docs/gguf.md
- llama.cpp: LLM inference in C/C++: https://github.com/ggml-org/llama.cpp
- Ollama documentation: OpenAI compatibility: https://docs.ollama.com/api/openai-compatibility
- An empirical study of LLaMA3 quantization: from LLMs to MLLMs: https://arxiv.org/abs/2404.14047
- Chat templates (Hugging Face Transformers documentation): https://huggingface.co/docs/transformers/chat_templating
- Pickle Scanning (Hugging Face Hub documentation): https://huggingface.co/docs/hub/security-pickle
- Exploiting LLM Quantization: https://arxiv.org/abs/2405.18137


---
Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.