GPU clusters & orchestration
Thousands of accelerators networked and scheduled to act as one machine for training or serving.
D1 · StoryFor young children and total beginners
One chip isn't enough to teach a big AI. So we connect thousands of chips together. They are like a huge orchestra that has to play in perfect time.
A conductor program decides which chips work on what. If a player stops, it starts the music again.
Like: An enormous orchestra with a conductor.
D2 · PictureFor curious beginners
Training a large model splits the work across many accelerators: each holds part of the data, part of the model, or both (data, tensor and pipeline parallelism).
The accelerators exchange results constantly, so fast networking between them is essential.
Schedulers such as Slurm or Kubernetes assign jobs to machines, and training jobs save checkpoints so a hardware failure doesn't lose weeks of work.
Running a large training job
- Submit: job request
- Schedule: allocate nodes
- Split: data / tensor / pipeline
- Sync: over the network
- Checkpoint: save progress
Like: An orchestra with a conductor keeping everyone in time.
Where it breaks: An orchestra survives a missed note. A synchronised training step waits for the slowest chip, so one failing machine can stall thousands.
D5 · FrontierFor experts
Open problems: keeping utilisation high on ever-larger clusters, training across multiple data centres, and scheduling training and inference workloads on shared hardware.