Layer 6 · Inference & Optimization

Running trained models quickly and cheaply for real users: serving, APIs, compression and caching.

In simple words: Getting answers out of a finished AI quickly and cheaply, for millions of people.

reviewed 2026-10-30

Like: A restaurant kitchen during service: the recipe is fixed, and now it's about getting many plates out fast and affordably.
Where it breaks: A kitchen cooks each order separately. Inference servers push many users' requests through the same model step together to use the hardware efficiently.

Key players

What it means for your business: This is where your per-request AI bill is set; a standard interface keeps switching providers cheap, and hosting open models yourself usually needs steady, high volume to pay off.

• vLLM: Open-source engine for serving models fast on GPUs; began at UC Berkeley.
• SGLang: Open-source serving engine used in production by labs and clouds.
• llama.cpp: Open-source engine that runs models on laptops and ordinary servers.
• Ollama: Free tool for running open models locally behind a simple API.
• Amazon Bedrock: AWS service that offers many companies' models through one API.
• OpenRouter: One API key that routes requests to many model providers.
• Together AI: Cloud for running, fine-tuning and hosting open models.

As of 2026-10-15. Sources: vLLM (GitHub) (2026); SGLang (GitHub) (2026); llama.cpp: LLM inference in C/C++ (GitHub) (2026); Ollama documentation: OpenAI compatibility (2024); Amazon Bedrock (2026); OpenRouter documentation: Quickstart (2024); Together AI (2026)

What it gives the layer above

A fast, affordable endpoint: send text in, get text out.

What it needs from the layer below

Model weights from Foundation Models and serving hardware from Cloud & Infrastructure.

Concepts

• Model APIs & routers
  Paying per token to send requests to a hosted model, either directly to a provider or through a router that picks among many providers.
• Batching & KV caching
  Serving tricks that process many users at once and avoid recomputing work already done.
• Quantization
  Storing a model's numbers with fewer bits so it's smaller and faster, at a small cost in accuracy.
• Local inference runtimes
  Programs that run downloaded model files on your own computer instead of a remote service.
• Speculative decoding
  A small, fast model guesses the next few tokens and the big model checks them all at once: fewer slow steps, with output identical under greedy decoding and the same distribution when sampling.

Walkthrough

1. A coding question goes out. A coding tool sends “why does this test fail?” to a model API, perhaps through a router that picks a provider, and pays per token.
2. The provider shares the work. On the provider's servers the question rides in a batch with many other users' requests, and the KV cache saves re-reading the long code file for every new token.
3. Smaller numbers, more room. Quantized weights take less memory, leaving room for more users' caches in each batch.
4. Or ask the same question at home. The same open model, quantized into a GGUF file, runs in a local runtime on a home GPU. The coding tool just points at localhost instead.
5. A small helper speeds it up. With only one user at home, a small draft model guesses ahead and the big model checks its guesses, so the answer streams out sooner: identical under greedy decoding, and with the same odds when sampling.

Examples

• Model APIs and routers
• Local runtimes and GGUF files
• Quantization
• Batching and KV caching
• Speculative decoding