Batching & KV caching
Serving tricks that process many users at once and avoid recomputing work already done.
Read first: Memory & bandwidth (HBM).
D1 · StoryFor young children and total beginners
A bus carries many people at once instead of one car per person. Batching sends many people's questions through the AI together.
The AI also keeps notes on what it has already read (a cache), so it doesn't re-read the whole conversation for every word.
Like: A bus instead of one car per person.
D2 · PictureFor curious beginners
Each generation step reads the model's weights; putting many requests in one step shares that expensive read across users.
Continuous batching adds and removes requests at every step, so short requests don't wait for long ones.
The KV cache stores intermediate results for tokens already processed, so each new token only does its own share of the work. It grows with context length and number of users, and its memory becomes the limit.
Serving many users at once
- Queue: incoming requests
- Batch step: one read of weights
- KV cache: reuse past work
- Tokens out: one per request
Like: A bus instead of one car per person.
Where it breaks: A bus waits for passengers to board. Continuous batching lets riders get on and off at every step, and each rider's luggage (their KV cache) takes space that limits how many fit.
D3 · PlayFor motivated learners
Each step of generation reads the model's weights once. If many users share that step, the expensive read is shared too, so the total output rises almost in step with the batch.
Each user still waits for the step, and each user's cache also has to be read. Slide the batch size and watch total speed and per-user speed separately.
Try the interactive version in the map.
Things to try
- With 250 tokens per conversation, go from 1 to 32 users. How much does each user's speed drop?
- Keep going to 256 users. What stops the total from rising?
- Switch to 4,000 tokens per conversation. How many users fit before memory runs out?
D4 · MechanismFor practitioners
The KV cache stores a key and a value for every past token, in every layer, so the model does not recompute them. Its size is 2 × layers × KV heads × head size × bytes × tokens × users.
The weights are a fixed cost. The cache grows with every token of context and every user, so on a busy server it soon overtakes the weights and becomes the memory budget.
Try the interactive version in the map.
Things to try
- With the 8B-class model, find a context length and user count where the cache is bigger than the weights.
- Double the context, then double the users instead. Do they cost the same?
- Switch the cache to 8-bit. How many more users fit in the same memory?
kv_per_token = 2 * layers * kv_heads * head_size * bytes # a key and a value per layer
cache = kv_per_token * context_tokens * users
weights = params * weight_bits / 8
free = device_memory - weights
max_users = free // (kv_per_token * context_tokens)Trade-offs
- Longer contexts and more users both grow the cache in direct proportion.
- Fewer KV heads (grouped-query attention) shrink the cache but change the model.
- A lower-precision cache saves memory at some risk to accuracy.
- Paging the cache in small blocks wastes less memory, and sharing common prefixes avoids storing the same tokens twice.
D5 · FrontierFor experts
Frontier: paged KV-cache memory management, sharing cached prefixes across users, running the prompt-reading and generation phases on different machines, and compressing the KV cache for very long contexts.