Model APIs & routers

D1 · StoryFor young children and total beginners

Guess first: what does the router do when one provider is down?

Most people never run an AI themselves. Their app sends a message over the internet to a company that runs it, and pays a tiny amount for each word.

A router is like a taxi app. You ask for a ride, and it picks which company sends the car.

Like a taxi app for AI

1. Your app: sends a message
2. The router: picks which company
3. AI company: runs the model
4. Answer back: plus a tiny bill

Like: A taxi app that picks which company sends the car.

D2 · PictureFor curious beginners

A model API is a web endpoint: you send the conversation as JSON and get the reply back, often streamed token by token. You pay per token, usually quoted per million, and input and output tokens are often priced differently.

Many providers and local servers copy the same request format, often called OpenAI-compatible. Switching can be as small as changing the base URL and the model name.

A router or aggregator sits in front of many providers. One key and one bill reach many models, and it can send each request to whichever provider is cheaper, faster or currently up.

Like: A taxi app that picks which company sends the car.
Where it breaks: Any taxi gets you to the same place. Two providers serving the "same" model can differ in quantization, context limit and supported features, so the answer you get may depend on which one the router picked.

D4 · MechanismFor practitioners

A request's bill is tokens × price, summed over its parts. Input and output are priced separately per million tokens, and output usually costs several times more per token than input.

Prompt caching stores a repeated prefix, such as instructions, tool definitions or a long document. A request that finds it cached pays a fraction of the input price for those tokens; one that misses pays a premium to write it. Only an exact prefix matches, so anything that changes must go at the end.

A batch interface trades speed for price: you submit many requests at once and collect the results within hours, at a discount. Agents multiply everything, because one task means many calls, each resending its context.

Try the interactive version in the map.

Things to try

• With chat and the cacheable prefix at 0%, find which part of the bill is biggest. Then raise output tokens to 4,000.
• Set the prefix to 90%, the hit rate to 25% and the write price to 2×. Is caching still saving money?
• Turn on batch. Does the share of the cost that comes from output change?
• Switch to multi-agent at 10,000 requests a day. What is the monthly bill, and how much of it does caching save?

# rates in $ per million tokens (illustrative)
tokens_in  = input_tokens  * workload   # chat 1, agent ~4, multi-agent ~15
tokens_out = output_tokens * workload
prefix = tokens_in * cached_share
fresh  = tokens_in - prefix
hits   = prefix * hit_rate
misses = prefix - hits
discount = 0.5 if batch else 1
cost = discount * (fresh  * in_price
                 + hits   * in_price * read_x     # e.g. 0.1
                 + misses * in_price * write_x    # e.g. 1.25
                 + tokens_out * out_price) / 1_000_000
per_day   = cost * requests_per_day
per_month = per_day * 30
# caching beats no caching when hit_rate * read_x + (1 - hit_rate) * write_x < 1

Trade-offs

• Caching pays only when the prefix is long, identical and reused before it expires. A cache write costs more than sending the same tokens uncached.
• Below a minimum length, from a few hundred to a few thousand tokens depending on the model, a prefix is not cached at all.
• A batch discount comes with a wait of up to a day, so it suits evaluations and bulk jobs, not chat.
• Output is the expensive side. Capping or shortening replies often saves more than squeezing the input.
• The agent and multi-agent multipliers are averages from one company's data. Measure your own, because the ratio depends on the task and the design.
• A per-token price is only part of the bill: retries, failed runs and growing contexts add tokens nobody planned for.

D5 · FrontierFor experts

One request format makes providers look interchangeable. In practice the same call can behave differently by provider, by date and even by server load.

Compatibility is a convention. Tool calling, caching, reasoning settings and errors differ between "compatible" servers, so a switch of provider needs its own tests (one compatible server).

Same name, different model. A hosted model can change behind a fixed name: one study saw accuracy on a maths task fall from 84% to 51% between two releases three months apart (Chen et al., 2023). Pin versions and rerun evals.

Not deterministic at temperature 0. Server kernels change their order of arithmetic with batch size, which depends on other users' traffic, so identical requests can get different answers (Thinking Machines, 2025).

Hidden tokens. Internal reasoning is billed as output tokens even when the app shows little of it, so the visible answer understates the cost (one provider’s docs).

Examples as of 2026-10-30

• Direct model APIs from OpenAI, Anthropic and Google
• Cloud model platforms: Amazon Bedrock, Google Vertex AI, Microsoft Foundry
• OpenRouter (router)
• vLLM and Ollama (local servers with OpenAI-compatible APIs)
• LiteLLM (open-source router)

Sources

1. OpenAI API reference: Chat Completions (2023)
2. OpenRouter documentation: Quickstart (2024)
3. vLLM documentation: OpenAI-Compatible Server (2025)
4. Pricing: prompt caching and batch processing (Claude API documentation) (2026-10-15)
5. Prompt caching (Claude API documentation) (2026-10-15)
6. Batch API guide (OpenAI API documentation) (2026-10-15)
7. Prompt caching guide (OpenAI API documentation) (2026-10-15)
8. How we built our multi-agent research system (Anthropic) (2025-06-13)
9. How is ChatGPT's behavior changing over time? (Chen, Zaharia, Zou) (2023-07-18)
10. Defeating Nondeterminism in LLM Inference (Thinking Machines Lab) (2025-09-10)
11. Extended thinking (Claude API documentation) (2026)

How it connects

• Local inference runtimes self-hosts instead of Model APIs & routers.
• Batching & KV caching lowers per-token price of Model APIs & routers.
• Model APIs & routers exposes function calling for Tool use & protocols (MCP).

More in Layer 6 · Inference & Optimization

• Batching & KV caching
  Serving tricks that process many users at once and avoid recomputing work already done.
• Quantization
  Storing a model's numbers with fewer bits so it's smaller and faster, at a small cost in accuracy.
• Local inference runtimes
  Programs that run downloaded model files on your own computer instead of a remote service.
• Speculative decoding
  A small, fast model guesses the next few tokens and the big model checks them all at once: fewer slow steps, with output identical under greedy decoding and the same distribution when sampling.