# Layer 6 · Inference & Optimization

URL: https://justuju.org/layer/inference-optimization/

Running trained models quickly and cheaply for real users: serving, APIs, compression and caching.

Gives the layer above: A fast, affordable endpoint: send text in, get text out.
Needs from the layer below: Model weights from Foundation Models and serving hardware from Cloud & Infrastructure.

## Concepts

- [Model APIs & routers](https://justuju.org/concept/model-apis/): Paying per token to send requests to a hosted model, either directly to a provider or through a router that picks among many providers.
- [Batching & KV caching](https://justuju.org/concept/batching-caching/): Serving tricks that process many users at once and avoid recomputing work already done.
- [Quantization](https://justuju.org/concept/quantization/): Storing a model's numbers with fewer bits so it's smaller and faster, at a small cost in accuracy.
- [Local inference runtimes](https://justuju.org/concept/local-inference/): Programs that run downloaded model files on your own computer instead of a remote service.
- [Speculative decoding](https://justuju.org/concept/speculative-decoding/): A small, fast model guesses the next few tokens and the big model checks them all at once: fewer slow steps, with output identical under greedy decoding and the same distribution when sampling.

---
Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.