Running locally vs in the cloud

D1 · StoryFor young children and total beginners

Guess first: what limits how big a model fits at home?

You can use an AI over the internet, on someone else's computers. Or you can download some AIs and run them on your own computer at home.

At home, your words stay with you. But your computer is much smaller, so the biggest AIs won't fit.

Like: Cooking at home versus eating at a restaurant.

D2 · PictureFor curious beginners

Cloud (an API or rented GPUs): the largest and newest models, no hardware to buy, pay per token or per hour, but your data leaves your network and the provider sets the rules.

Local (a desktop with one or more consumer GPUs, or a compact AI workstation): only open-weight models, limited mainly by how much fast memory you have, but private, works offline, no per-token bill, and you choose the model, including fine-tuned or less filtered ones.

Quantisation is what makes local practical: storing weights with fewer bits lets a bigger model fit in the same memory, at some cost in quality. A local runtime handles the download and running.

Like: Cooking at home versus eating at a restaurant.
Where it breaks: A home kitchen can cook most dishes a restaurant can. A home machine can't run the largest frontier models at all, and many of those are never released as weights, so local and cloud are often used side by side.

D5 · FrontierFor experts

D2 compares local and cloud on models, privacy and cost per token. Each of those comparisons moves quickly.

Falling token prices. The price of reaching a fixed level of performance fell between 9 and 900 times a year depending on the task, so buying hardware to save money is a moving target (Cottier et al., 2025).

Batching economics. Cloud servers share each weight read across many users; better cache management alone raised serving throughput 2 to 4 times (Kwon et al., 2023).

The open-model gap. Local setups run only open-weight models, which have lagged the best closed models by about a year on benchmarks (Cottier et al., 2024).

Privacy by design. Cloud services can promise stateless processing and no administrator access, and let outside researchers check it, which narrows the privacy gap if the checks hold (Apple, 2024).

Examples as of 2026-10-13

• llama.cpp and Ollama (local runtimes)
• Model APIs and rented GPUs (cloud)

Sources

1. llama.cpp: LLM inference in C/C++ (GitHub) (2026)
2. NVIDIA DGX Spark product page (2025)
3. LLM inference prices have fallen rapidly but unequally across tasks (Epoch AI) (2025-03-12)
4. Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)
5. How far behind are open models? (Cottier et al., Epoch AI) (2024-11-04)
6. Private Cloud Compute: A new frontier for AI privacy in the cloud (Apple Security Research) (2024-06-10)

How it connects

• Running locally vs in the cloud avoids renting from GPU clouds & neoclouds.

More in Layer 3 · Cloud & Infrastructure

• Cluster networking
  The very fast network that lets thousands of accelerators share results every training step, so the whole cluster behaves like one computer.
• GPU clusters & orchestration
  Thousands of accelerators networked and scheduled to act as one machine for training or serving.
• GPU clouds & neoclouds
  Companies that own huge GPU clusters and rent slices of them by the hour, from big general clouds to specialist GPU-only providers.
• Training vs inference infrastructure
  Training needs one giant, tightly linked cluster for weeks; serving answers needs many smaller copies close to users, sized for speed and cost per request.