JustujuAI, layer by layer.Open in the interactive map

Batching & KV caching

Serving tricks that process many users at once and avoid recomputing work already done.

Layer 6 · Inference & Optimization · Run time · reviewed 2026-10-30

Read first: Memory & bandwidth (HBM).

D1 · StoryFor young children and total beginners

Guess first: is a bus or a line of cars quicker for fifty people?

A bus carries many people at once instead of one car per person. Batching sends many people's questions through the AI together.

The AI also keeps notes on what it has already read (a cache), so it doesn't re-read the whole conversation for every word.

Like: A bus instead of one car per person.

D2 · PictureFor curious beginners

Each generation step reads the model's weights; putting many requests in one step shares that expensive read across users.

Continuous batching adds and removes requests at every step, so short requests don't wait for long ones.

The KV cache stores intermediate results for tokens already processed, so each new token only does its own share of the work. It grows with context length and number of users, and its memory becomes the limit.

Serving many users at once

  1. Queue: incoming requests
  2. Batch step: one read of weights
  3. KV cache: reuse past work
  4. Tokens out: one per request

Like: A bus instead of one car per person.
Where it breaks: A bus waits for passengers to board. Continuous batching lets riders get on and off at every step, and each rider's luggage (their KV cache) takes space that limits how many fit.

D3 · PlayFor motivated learners

Each step of generation reads the model's weights once. If many users share that step, the expensive read is shared too, so the total output rises almost in step with the batch.

Each user still waits for the step, and each user's cache also has to be read. Slide the batch size and watch total speed and per-user speed separately.

Try the interactive version in the map.

Things to try

D4 · MechanismFor practitioners

The KV cache stores a key and a value for every past token, in every layer, so the model does not recompute them. Its size is 2 × layers × KV heads × head size × bytes × tokens × users.

The weights are a fixed cost. The cache grows with every token of context and every user, so on a busy server it soon overtakes the weights and becomes the memory budget.

Try the interactive version in the map.

Things to try

kv_per_token = 2 * layers * kv_heads * head_size * bytes   # a key and a value per layer
cache = kv_per_token * context_tokens * users
weights = params * weight_bits / 8
free = device_memory - weights
max_users = free // (kv_per_token * context_tokens)

Trade-offs

D5 · FrontierFor experts

Batching looks like free throughput, but the real limit is memory: every running request keeps a KV cache that grows with its length.

Fragmentation. Reserving cache space for the longest possible answer wastes memory; managing it in small pages raised throughput 2 to 4 times at the same latency (PagedAttention, 2023).

Prefill and decode interfere. Reading a prompt is compute-bound and generating is memory-bound. Running them on separate GPUs served up to 7.4 times more requests within latency targets in one study, but the right split depends on the workload (DistServe, 2024).

Sharing prefixes. Reusing cache for a shared beginning, such as a system prompt, saves work, but deciding which prefixes to keep in limited memory is a scheduling problem (SGLang, 2023).

Long contexts. At long lengths the cache can outgrow the weights. Storing it in 2 bits cut peak memory 2.6 times with little quality loss in one study; how far compression can go before long-range recall suffers is open (KIVI, 2024).

Examples as of 2026-10-30

Sources

  1. Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)
  2. Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI) (2022)
  3. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (2024-01-18)
  4. SGLang: Efficient Execution of Structured Language Model Programs (2023-12-12)
  5. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024-02-05)

How it connects

More in Layer 6 · Inference & Optimization