# Training vs inference infrastructure URL: https://justuju.org/concept/training-vs-inference/ Layer 3 · Cloud & Infrastructure Training needs one giant, tightly linked cluster for weeks; serving answers needs many smaller copies close to users, sized for speed and cost per request. ## D1 · Story Teaching an AI is like building a giant ship: thousands of workers on one huge job for weeks. Using the AI is more like running a chain of cafés. Lots of small shops serve people quickly, all day, near where they live. Like: Building one ship versus running a chain of cafés. ## D2 · Picture Training runs one job across thousands of accelerators in one place. They must swap results every step, so the fast network between them matters as much as the chips. Jobs last weeks and the goal is high utilisation. Inference serves many separate requests. Each copy of the model needs only enough chips to hold its weights and working memory, and copies can sit in different regions. The goals are low latency and low cost per token, and demand rises and falls through the day. Inference itself has two phases: reading the prompt (prefill) is compute-heavy, while writing the answer token by token (decode) is limited by memory bandwidth. Some systems now run the two on different machines. Like: Building one ship versus running a chain of cafés. Where it breaks: Cafés and shipyards need different buildings. GPU clusters are more flexible: the same machines can switch between training and serving, and growing amounts of compute now go into inference, for example when models reason for longer before answering. ## D5 · Frontier D2 separates training and inference by the shape of the job. Newer methods mix the two and move spending between them. Reinforcement learning mixes them. Post-training with RL alternates generating answers and updating weights, so systems now reshard one model between training and generation on shared hardware (Sheng et al., 2024). Splitting inference. Running prompt reading and answer writing on separate GPUs removes interference between them, at the cost of moving the cache; how far to split is unsettled (Zhong et al., 2024). Train or think. On problems a smaller model can sometimes solve, extra thinking at answer time beat a 14 times larger model at matched compute, so the split of a budget between training and inference is a design choice (Snell et al., 2024). ## Examples as of 2026-10-30 - Training clusters: xAI Colossus, Meta's Llama training clusters - Inference-focused chips: Google TPU Ironwood, AWS Inferentia, Cerebras - verl (open framework that mixes RL training and generation) ## Sources - Splitwise: Efficient generative LLM inference using phase splitting: https://arxiv.org/abs/2311.18677 - Efficient Memory Management for Large Language Model Serving with PagedAttention: https://arxiv.org/abs/2309.06180 - Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism: https://arxiv.org/abs/1909.08053 - HybridFlow: A Flexible and Efficient RLHF Framework (Sheng et al.): https://arxiv.org/abs/2409.19256 - DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (Zhong et al.): https://arxiv.org/abs/2401.09670 - Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters: https://arxiv.org/abs/2408.03314 - verl: Volcano Engine Reinforcement Learning for LLMs (GitHub): https://github.com/verl-project/verl --- Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.