Ring All-Reduce: GPU Cluster Gradient Sync

Cluster Setup

GPUs in the ring 8

Model size 7 B params

Interconnect bandwidth 300 GB/s

Per-GPU compute (TFLOP/s) 400

Step Timing

compute 50%comm 50%

Compute time / step

–

All-reduce time / step

–

GPU utilization

–

Steps completed

How it works

Frontier models train across many GPUs in parallel (data parallelism). After every micro-batch each GPU has computed its own gradient; before the next step all GPUs need the averaged gradient. Ring all-reduce does this in two passes around a ring of N GPUs, each holding a 1/N chunk of the gradient buffer, with no central server:

Scatter-reduce: N-1 steps, each GPU sends 1/N of its buffer to its neighbor
                and adds the chunk it receives — after N-1 steps every GPU
                holds one fully-reduced 1/N chunk.
All-gather:     N-1 more steps circulate those reduced chunks so every
                GPU ends with the complete averaged gradient.

Data moved per GPU  ≈ 2·(N-1)/N · S   (S = gradient buffer size in bytes)
Comm time            = data moved / bandwidth
Compute time          = FLOPs per step / per-GPU TFLOP/s

GPUs in the ring — more GPUs means more parallel compute, but also more communication hops; the per-GPU data moved barely shrinks once N is large (it saturates near 2S).
Model size — sets the gradient buffer S (≈ 4 bytes × parameter count in fp32) that has to be synchronized every step.
Interconnect bandwidth — NVLink/InfiniBand speed between GPUs; this is exactly why frontier labs build custom low-latency fabrics instead of using commodity Ethernet.
Per-GPU compute — how fast each accelerator finishes its local forward/backward pass.
Run Training Step — animates one scatter-reduce + all-gather cycle around the ring and updates the compute-vs-communication split; when the red "comm" bar dominates, the cluster is communication-bound — adding more GPUs stops helping until bandwidth improves, which is the real engineering constraint behind compute-optimal scaling laws.

Real-world relevance: this is the mechanism (NCCL ring/tree all-reduce) inside every large-scale PyTorch/JAX training run behind GPT-4-class and Gemini-class models — scaling laws describe how much compute buys how much loss reduction, but ring all-reduce is what determines whether that compute is actually reachable at a given cluster size.

GPU node Reduced gradient chunk In-flight chunk (ring link)

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.