Ring All-Reduce 2D: Bandwidth vs Latency

Cluster Setup

GPUs in the ring 8

Model size 70 B params

Interconnect bandwidth 300 GB/s

Per-GPU compute (TFLOP/s) 400

Per-hop latency 20 µs

Step Timing

compute 50%comm 50%

Compute time / step

–

All-reduce time / step

–

GPU utilization

–

Steps completed

How it works

This is the flat 2D counterpart of the 3D ring all-reduce simulator: the same distributed-training mechanism, redrawn as a top-down ring diagram plus a scrolling timing strip chart. Every GPU holds a 1/N slice of the gradient buffer; ring all-reduce syncs it in two passes with no central server:

Scatter-reduce: N-1 steps, each GPU sends 1/N of its buffer to its neighbor
                and adds the chunk it receives — after N-1 steps every GPU
                holds one fully-reduced 1/N chunk.
All-gather:     N-1 more steps circulate those reduced chunks so every
                GPU ends with the complete averaged gradient.

Data moved per GPU  ≈ 2·(N-1)/N · S        (S = gradient buffer size)
Bandwidth time       = data moved / bandwidth
Latency overhead     = 2·(N-1) hops · per-hop latency   (added in this 2D build)
Comm time            = bandwidth time + latency overhead
Compute time          = FLOPs per step / per-GPU TFLOP/s

GPUs in the ring — more GPUs adds parallel compute but also more hops; per-GPU data moved saturates near 2S as N grows.
Model size — sets the gradient buffer S (≈ 4 bytes × parameter count, fp32) synchronized every step.
Interconnect bandwidth — raw NVLink/InfiniBand throughput between neighbors.
Per-GPU compute — local forward/backward throughput per accelerator.
Per-hop latency — fixed per-message overhead (kernel launch + wire delay) that the 3D sibling omits; at small buffers or huge ring sizes this fixed cost, not bandwidth, dominates the comm time — drag the GPU count to 24 with a small model to see the latency floor.
Drag the ring — click-drag the ring diagram to rotate it; the physics does not change, only the viewing angle, mirroring the orbit control on the 3D version.
Run Training Step — animates one scatter-reduce + all-gather cycle and logs a bar to the timeline strip chart below the ring.

Real-world relevance: this is the mechanism (NCCL ring/tree all-reduce) inside every large-scale PyTorch/JAX training run behind frontier-scale models — when the red "comm" bar dominates, the cluster is communication-bound and adding GPUs stops helping until bandwidth or latency improves.

GPU node Reduced gradient chunk In-flight chunk (ring link)

drag to rotate

Compute vs comm — last 40 steps