Training vs inference infrastructure
Training needs one giant, tightly linked cluster for weeks; serving answers needs many smaller copies close to users, sized for speed and cost per request.
Read first: GPU clusters & orchestration.
D1 · StoryFor young children and total beginners
Teaching an AI is like building a giant ship: thousands of workers on one huge job for weeks.
Using the AI is more like running a chain of cafés. Lots of small shops serve people quickly, all day, near where they live.
Like: Building one ship versus running a chain of cafés.
D2 · PictureFor curious beginners
Training runs one job across thousands of accelerators in one place. They must swap results every step, so the fast network between them matters as much as the chips. Jobs last weeks and the goal is high utilisation.
Inference serves many separate requests. Each copy of the model needs only enough chips to hold its weights and working memory, and copies can sit in different regions. The goals are low latency and low cost per token, and demand rises and falls through the day.
Inference itself has two phases: reading the prompt (prefill) is compute-heavy, while writing the answer token by token (decode) is limited by memory bandwidth. Some systems now run the two on different machines.
Like: Building one ship versus running a chain of cafés.
Where it breaks: Cafés and shipyards need different buildings. GPU clusters are more flexible: the same machines can switch between training and serving, and growing amounts of compute now go into inference, for example when models reason for longer before answering.
D5 · FrontierFor experts
The line is blurring. Reinforcement learning in post-training mixes the two, generating lots of answers (inference) inside a training loop.
Reasoning models and agents shift spending towards inference, and it is an open question how to share one fleet between both kinds of work without either waiting.