# Batching & KV caching URL: https://justuju.org/concept/batching-caching/ Layer 6 · Inference & Optimization Serving tricks that process many users at once and avoid recomputing work already done. ## D1 · Story A bus carries many people at once instead of one car per person. Batching sends many people's questions through the AI together. The AI also keeps notes on what it has already read (a cache), so it doesn't re-read the whole conversation for every word. Like: A bus instead of one car per person. ## D2 · Picture Each generation step reads the model's weights; putting many requests in one step shares that expensive read across users. Continuous batching adds and removes requests at every step, so short requests don't wait for long ones. The KV cache stores intermediate results for tokens already processed, so each new token only does its own share of the work. It grows with context length and number of users, and its memory becomes the limit. 1. Queue: incoming requests 2. Batch step: one read of weights 3. KV cache: reuse past work 4. Tokens out: one per request Like: A bus instead of one car per person. Where it breaks: A bus waits for passengers to board. Continuous batching lets riders get on and off at every step, and each rider's luggage (their KV cache) takes space that limits how many fit. ## D4 · Mechanism The KV cache stores a key and a value for every past token, in every layer, so the model does not recompute them. Its size is 2 × layers × KV heads × head size × bytes × tokens × users. The weights are a fixed cost. The cache grows with every token of context and every user, so on a busy server it soon overtakes the weights and becomes the memory budget. - Longer contexts and more users both grow the cache in direct proportion. - Fewer KV heads (grouped-query attention) shrink the cache but change the model. - A lower-precision cache saves memory at some risk to accuracy. - Paging the cache in small blocks wastes less memory, and sharing common prefixes avoids storing the same tokens twice. ## D5 · Frontier Batching looks like free throughput, but the real limit is memory: every running request keeps a KV cache that grows with its length. Fragmentation. Reserving cache space for the longest possible answer wastes memory; managing it in small pages raised throughput 2 to 4 times at the same latency (PagedAttention, 2023). Prefill and decode interfere. Reading a prompt is compute-bound and generating is memory-bound. Running them on separate GPUs served up to 7.4 times more requests within latency targets in one study, but the right split depends on the workload (DistServe, 2024). Sharing prefixes. Reusing cache for a shared beginning, such as a system prompt, saves work, but deciding which prefixes to keep in limited memory is a scheduling problem (SGLang, 2023). Long contexts. At long lengths the cache can outgrow the weights. Storing it in 2 bits cut peak memory 2.6 times with little quality loss in one study; how far compression can go before long-range recall suffers is open (KIVI, 2024). ## Examples as of 2026-10-30 - vLLM (PagedAttention) - SGLang (RadixAttention prefix cache) - Prompt caching in the Anthropic, OpenAI and Google APIs - NVIDIA TensorRT-LLM and Dynamo ## Sources - Efficient Memory Management for Large Language Model Serving with PagedAttention: https://arxiv.org/abs/2309.06180 - Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI): https://www.usenix.org/conference/osdi22/presentation/yu - DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving: https://arxiv.org/abs/2401.09670 - SGLang: Efficient Execution of Structured Language Model Programs: https://arxiv.org/abs/2312.07104 - KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache: https://arxiv.org/abs/2402.02750 --- Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.