GPU clusters & orchestration

D1 · StoryFor young children and total beginners

What happens to an orchestra if one player stops? Keep that in mind.

One chip isn't enough to teach a big AI. So we connect thousands of chips together. They are like a huge orchestra that has to play in perfect time.

A conductor program decides which chips work on what. If a player stops, it starts the music again.

The conductor program runs one huge job

1. One big job: teach a big AI
2. Split it up: across thousands of chips
3. Play in time: chips share results
4. Save progress: every so often

Like: An enormous orchestra with a conductor.

D2 · PictureFor curious beginners

Training a large model splits the work across many accelerators: each holds part of the data, part of the model, or both (data, tensor and pipeline parallelism).

The accelerators exchange results constantly, so fast networking between them is essential.

Schedulers such as Slurm or Kubernetes assign jobs to machines, and training jobs save checkpoints so a hardware failure doesn't lose weeks of work.

Running a large training job

1. Submit: job request
2. Schedule: allocate nodes
3. Split: data / tensor / pipeline
4. Sync: over the network
5. Checkpoint: save progress

Like: An orchestra with a conductor keeping everyone in time.
Where it breaks: An orchestra survives a missed note. A synchronised training step waits for the slowest chip, so one failing machine can stall thousands.

D4 · MechanismFor practitioners

Training needs far more memory than the weights alone. With 16-bit weights and the Adam optimizer, each parameter carries about 16 bytes: 2 for the weight, 2 for its gradient and 12 for the optimizer's 32-bit copy and running averages. A 70B model needs about 1.1 TB before counting activations.

There are three ways to split the work. Tensor parallelism cuts each layer's matrices across accelerators. Pipeline parallelism gives each accelerator a run of layers, and passes micro-batches along like an assembly line. Data parallelism runs copies that each see different data and average their gradients; the copies can also shard the optimizer state, gradients and weights between them instead of each keeping a full set.

The total is tensor × pipeline × data-parallel accelerators. Pipelines waste time while they fill and drain (the bubble), so they need many micro-batches. Total training compute is about 6 × parameters × tokens, which sets how long the whole cluster takes.

Try the interactive version in the map.

Things to try

• Start with the 70B model on one 80 GB accelerator. Raise tensor and pipeline ways until it fits. How few accelerators can hold one copy?
• Set pipeline stages to 8 and move the micro-batch slider from 1 up to 128. What happens to the idle share?
• Go back to no splitting and choose "Everything" for sharding. How many data-parallel copies does it take to fit without splitting the model?
• Double the data-parallel copies a few times. What happens to the days, and what does each doubling cost?

bytes_per_param = 2 + 2 + 12                  # 16-bit weight, gradient, Adam state (32-bit copy, momentum, variance)
# sharding across dp copies divides: optimizer only -> 4 + 12/dp, + gradients -> 2 + 14/dp, everything -> 16/dp
memory_each = params * bytes_per_param / (tp * pp)   # plus activations, not counted here
accelerators = tp * pp * dp
bubble = (pp - 1) / (micro_batches + pp - 1)  # share of each step a pipeline sits idle
train_flops = 6 * params * tokens             # forward and backward pass
days = train_flops / (accelerators * peak_flops * utilisation * (1 - bubble)) / 86400

Trade-offs

• Tensor parallelism talks to other accelerators inside every layer, so it is kept to fast links within one machine.
• Pipeline stages talk less, but each stage idles while the pipeline fills and drains; more micro-batches shrink that bubble.
• Sharding saves memory but adds network traffic to gather weights and spread gradients at every step.
• More data-parallel copies cut the days almost in proportion, until the batch gets too large to train well or the network becomes the limit.

D5 · FrontierFor experts

D4's arithmetic assumes every accelerator is healthy, equally fast and always busy. At scale none of that holds.

Utilisation. A recent large run reported 38% to 43% model FLOPs utilisation on 16,000 GPUs, so more than half of peak arithmetic went unused (Grattafiori et al., 2024).

Reliability. Eleven months and over 150 million GPU-hours on two research clusters showed the largest jobs most exposed to failure; the authors call for reliability-aware systems and algorithms (Kokolis et al., 2024).

Cost. The amortised cost of the most compute-intensive training runs has grown about 2.4 times a year since 2016. On that trend the largest pass $1 billion by 2027, raising the question of who can afford frontier training (Cottier et al., 2024).

Examples as of 2026-10-30

• Slurm (job scheduler)
• Kubernetes (container orchestration)
• Ray (distributed Python)
• Megatron-LM and PyTorch FSDP (distributed training)

Sources

1. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019)
2. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM (2021-04-09)
3. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (2019-10-04)
4. Scaling Laws for Neural Language Models (2020-01-23)
5. The Llama 3 Herd of Models (section on training infrastructure) (2024)
6. Revisiting Reliability in Large-Scale Machine Learning Research Clusters (Kokolis et al.) (2024-10-29)
7. The rising costs of training frontier AI models (Cottier et al.) (2024-05-31)

How it connects

• Power & cooling for AI data centres powers and cools GPU clusters & orchestration.
• GPUs & AI accelerators networked into GPU clusters & orchestration.
• Cluster networking ties together GPU clusters & orchestration.
• GPU clusters & orchestration are rented out by GPU clouds & neoclouds.
• Chip-to-chip interconnect joins chips into GPU clusters & orchestration.
• GPU clusters & orchestration runs classifiers for Data curation and mixing.

More in Layer 3 · Cloud & Infrastructure

• Cluster networking
  The very fast network that lets thousands of accelerators share results every training step, so the whole cluster behaves like one computer.
• GPU clouds & neoclouds
  Companies that own huge GPU clusters and rent slices of them by the hour, from big general clouds to specialist GPU-only providers.
• Training vs inference infrastructure
  Training needs one giant, tightly linked cluster for weeks; serving answers needs many smaller copies close to users, sized for speed and cost per request.
• Running locally vs in the cloud
  Open-weight models can run on your own hardware at home, trading the cloud's scale and convenience for privacy, control and a fixed cost.