# Model APIs & routers URL: https://justuju.org/concept/model-apis/ Layer 6 · Inference & Optimization Paying per token to send requests to a hosted model, either directly to a provider or through a router that picks among many providers. ## D1 · Story Most people never run an AI themselves. Their app sends a message over the internet to a company that runs it, and pays a tiny amount for each word. A router is like a taxi app. You ask for a ride, and it picks which company sends the car. 1. Your app: sends a message 2. The router: picks which company 3. AI company: runs the model 4. Answer back: plus a tiny bill Like: A taxi app that picks which company sends the car. ## D2 · Picture A model API is a web endpoint: you send the conversation as JSON and get the reply back, often streamed token by token. You pay per token, usually quoted per million, and input and output tokens are often priced differently. Many providers and local servers copy the same request format, often called OpenAI-compatible. Switching can be as small as changing the base URL and the model name. A router or aggregator sits in front of many providers. One key and one bill reach many models, and it can send each request to whichever provider is cheaper, faster or currently up. Like: A taxi app that picks which company sends the car. Where it breaks: Any taxi gets you to the same place. Two providers serving the "same" model can differ in quantization, context limit and supported features, so the answer you get may depend on which one the router picked. ## D4 · Mechanism A request's bill is tokens × price, summed over its parts. Input and output are priced separately per million tokens, and output usually costs several times more per token than input. Prompt caching stores a repeated prefix, such as instructions, tool definitions or a long document. A request that finds it cached pays a fraction of the input price for those tokens; one that misses pays a premium to write it. Only an exact prefix matches, so anything that changes must go at the end. A batch interface trades speed for price: you submit many requests at once and collect the results within hours, at a discount. Agents multiply everything, because one task means many calls, each resending its context. - Caching pays only when the prefix is long, identical and reused before it expires. A cache write costs more than sending the same tokens uncached. - Below a minimum length, from a few hundred to a few thousand tokens depending on the model, a prefix is not cached at all. - A batch discount comes with a wait of up to a day, so it suits evaluations and bulk jobs, not chat. - Output is the expensive side. Capping or shortening replies often saves more than squeezing the input. - The agent and multi-agent multipliers are averages from one company's data. Measure your own, because the ratio depends on the task and the design. - A per-token price is only part of the bill: retries, failed runs and growing contexts add tokens nobody planned for. ## D5 · Frontier One request format makes providers look interchangeable. In practice the same call can behave differently by provider, by date and even by server load. Compatibility is a convention. Tool calling, caching, reasoning settings and errors differ between "compatible" servers, so a switch of provider needs its own tests (one compatible server). Same name, different model. A hosted model can change behind a fixed name: one study saw accuracy on a maths task fall from 84% to 51% between two releases three months apart (Chen et al., 2023). Pin versions and rerun evals. Not deterministic at temperature 0. Server kernels change their order of arithmetic with batch size, which depends on other users' traffic, so identical requests can get different answers (Thinking Machines, 2025). Hidden tokens. Internal reasoning is billed as output tokens even when the app shows little of it, so the visible answer understates the cost (one provider’s docs). ## Examples as of 2026-10-30 - Direct model APIs from OpenAI, Anthropic and Google - Cloud model platforms: Amazon Bedrock, Google Vertex AI, Microsoft Foundry - OpenRouter (router) - vLLM and Ollama (local servers with OpenAI-compatible APIs) - LiteLLM (open-source router) ## Sources - OpenAI API reference: Chat Completions: https://developers.openai.com/api/reference/resources/chat/ - OpenRouter documentation: Quickstart: https://openrouter.ai/docs/quickstart - vLLM documentation: OpenAI-Compatible Server: https://docs.vllm.ai/en/v0.8.3/serving/openai_compatible_server.html - Pricing: prompt caching and batch processing (Claude API documentation): https://platform.claude.com/docs/en/about-claude/pricing - Prompt caching (Claude API documentation): https://platform.claude.com/docs/en/build-with-claude/prompt-caching - Batch API guide (OpenAI API documentation): https://developers.openai.com/api/docs/guides/batch - Prompt caching guide (OpenAI API documentation): https://developers.openai.com/api/docs/guides/prompt-caching - How we built our multi-agent research system (Anthropic): https://www.anthropic.com/engineering/multi-agent-research-system - How is ChatGPT's behavior changing over time? (Chen, Zaharia, Zou): https://arxiv.org/abs/2307.09009 - Defeating Nondeterminism in LLM Inference (Thinking Machines Lab): https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ - Extended thinking (Claude API documentation): https://platform.claude.com/docs/en/build-with-claude/extended-thinking --- Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.