Training vs inference infrastructure

D1 · StoryFor young children and total beginners

Look at the two columns: which one needs every chip talking at once?

Teaching an AI is like building a giant ship: thousands of workers on one huge job for weeks.

Using the AI is more like running a chain of cafés. Lots of small shops serve people quickly, all day, near where they live.

Like: Building one ship versus running a chain of cafés.

D2 · PictureFor curious beginners

Training runs one job across thousands of accelerators in one place. They must swap results every step, so the fast network between them matters as much as the chips. Jobs last weeks and the goal is high utilisation.

Inference serves many separate requests. Each copy of the model needs only enough chips to hold its weights and working memory, and copies can sit in different regions. The goals are low latency and low cost per token, and demand rises and falls through the day.

Inference itself has two phases: reading the prompt (prefill) is compute-heavy, while writing the answer token by token (decode) is limited by memory bandwidth. Some systems now run the two on different machines.

Like: Building one ship versus running a chain of cafés.
Where it breaks: Cafés and shipyards need different buildings. GPU clusters are more flexible: the same machines can switch between training and serving, and growing amounts of compute now go into inference, for example when models reason for longer before answering.

D5 · FrontierFor experts

D2 separates training and inference by the shape of the job. Newer methods mix the two and move spending between them.

Reinforcement learning mixes them. Post-training with RL alternates generating answers and updating weights, so systems now reshard one model between training and generation on shared hardware (Sheng et al., 2024).

Splitting inference. Running prompt reading and answer writing on separate GPUs removes interference between them, at the cost of moving the cache; how far to split is unsettled (Zhong et al., 2024).

Train or think. On problems a smaller model can sometimes solve, extra thinking at answer time beat a 14 times larger model at matched compute, so the split of a budget between training and inference is a design choice (Snell et al., 2024).

Examples as of 2026-10-30

• Training clusters: xAI Colossus, Meta's Llama training clusters
• Inference-focused chips: Google TPU Ironwood, AWS Inferentia, Cerebras
• verl (open framework that mixes RL training and generation)

Sources

1. Splitwise: Efficient generative LLM inference using phase splitting (2023)
2. Efficient Memory Management for Large Language Model Serving with PagedAttention (2023)
3. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019)
4. HybridFlow: A Flexible and Efficient RLHF Framework (Sheng et al.) (2024-09-28)
5. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (Zhong et al.) (2024-01-18)
6. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (2024)
7. verl: Volcano Engine Reinforcement Learning for LLMs (GitHub) (2026)

How it connects

• GPU clouds & neoclouds rents capacity for Training vs inference infrastructure.

More in Layer 3 · Cloud & Infrastructure

• Cluster networking
  The very fast network that lets thousands of accelerators share results every training step, so the whole cluster behaves like one computer.
• GPU clusters & orchestration
  Thousands of accelerators networked and scheduled to act as one machine for training or serving.
• GPU clouds & neoclouds
  Companies that own huge GPU clusters and rent slices of them by the hour, from big general clouds to specialist GPU-only providers.
• Running locally vs in the cloud
  Open-weight models can run on your own hardware at home, trading the cloud's scale and convenience for privacy, control and a fixed cost.