Local inference runtimes

D1 · StoryFor young children and total beginners

Look at the stack: which layer decides whether the model fits?

Some AI models can be downloaded as one big file, like a film you keep instead of stream.

A local runtime is the player for that file. It runs the model on your own computer, so your words never leave the house.

The catch: the file has to fit in your computer's memory.

Like: Downloading a film instead of streaming it.

D2 · PictureFor curious beginners

A local runtime loads a model's weights into your GPU or CPU memory and generates tokens on your machine.

Models are usually shared as single files that bundle the weights, the tokenizer and settings such as the chat template. GGUF is a common format, and each model is often published in several quantization levels.

Most runtimes also start a small local server that speaks the same request format as the big cloud APIs, so existing apps and coding tools can point at your machine instead.

Memory decides what you can run: the weights (parameters × bits ÷ 8) plus the KV cache for your context must fit. Memory bandwidth then sets how fast tokens come out.

Like: Downloading a film instead of streaming it.
Where it breaks: A downloaded film plays the same on any player. A model file can run noticeably differently depending on the quantization level, the runtime's settings and whether the chat template matches, and a big model may not fit at all.

D5 · FrontierFor experts

"Does it fit in memory?" is only the first question. Speed, quality and trust depend on details the rule of thumb hides.

Splitting a model. Runtimes can put some layers on the CPU when the GPU is full, so a model too big for the GPU still runs, but tokens then come out at the pace of the slower memory (llama.cpp).

Quality per bit varies. The same bit width costs some models more than others; one study found a newer model degraded noticeably at low bit widths (2024 study). Community testing is often the only guide.

Templates and settings. A wrong chat template or missing control tokens can make a model "drastically worse" with no error message (chat template docs), so a local model can trail the same model in the cloud.

Trusting downloaded files. Pickle-based files can run code when loaded (pickle scanning docs), and a model can be crafted to misbehave only after quantization (Egashira et al., 2024). Where a file came from matters.

Examples as of 2026-10-13

• llama.cpp
• Ollama
• LM Studio

Sources

1. GGUF file format specification (ggml) (2023)
2. llama.cpp: LLM inference in C/C++ (2023)
3. Ollama documentation: OpenAI compatibility (2024)
4. An empirical study of LLaMA3 quantization: from LLMs to MLLMs (2024-04-22)
5. Chat templates (Hugging Face Transformers documentation) (ongoing)
6. Pickle Scanning (Hugging Face Hub documentation) (ongoing)
7. Exploiting LLM Quantization (2024-05-28)

How it connects

• Speculative decoding speeds single-user generation in Local inference runtimes.
• Quantization fits bigger models into Local inference runtimes.
• Local inference runtimes runs downloaded files of Open vs closed weights.
• Local inference runtimes runs on Local AI hardware.
• Local inference runtimes self-hosts instead of Model APIs & routers.

More in Layer 6 · Inference & Optimization

• Model APIs & routers
  Paying per token to send requests to a hosted model, either directly to a provider or through a router that picks among many providers.
• Batching & KV caching
  Serving tricks that process many users at once and avoid recomputing work already done.
• Quantization
  Storing a model's numbers with fewer bits so it's smaller and faster, at a small cost in accuracy.
• Speculative decoding
  A small, fast model guesses the next few tokens and the big model checks them all at once: fewer slow steps, with output identical under greedy decoding and the same distribution when sampling.