Cluster networking
The very fast network that lets thousands of accelerators share results every training step, so the whole cluster behaves like one computer.
Read first: GPU clusters & orchestration.
D1 · StoryFor young children and total beginners
When thousands of chips learn together, they must keep sharing notes, many times a second.
So they are joined by super fast cables, like a team passing a ball. If one pass is slow, the whole team waits.
Like: A team passing a ball: one slow pass holds everyone up.
D2 · PictureFor curious beginners
Inside one server, accelerators talk over a very fast local link. Between servers, a fabric of switches connects them, usually InfiniBand or high-speed Ethernet with RDMA, which moves data straight between memories without the CPU in the way.
After each training step, every accelerator must combine its results with all the others (a collective such as all-reduce). The step can't finish until the slowest transfer does.
That is why people say the network is the computer: adding chips only helps if the network can keep them fed, and network layout decides which jobs can run where.
Like: A team passing a ball: one slow pass holds everyone up.
Where it breaks: A team can improvise around a dropped pass. A synchronised collective can't: one faulty link or congested switch slows every chip in the job, and finding which of thousands of links is to blame is hard.
D5 · FrontierFor experts
Open problems: training across several data centres where links are slower, keeping huge fabrics free of congestion, and spotting failing links before they stall a job.
InfiniBand and Ethernet compete for AI clusters, and industry groups are extending Ethernet for this workload.