Home▸Machine Learning & Neural Networks▸Ring All-Reduce 2D: Bandwidth vs Latency

Ring All-Reduce 2D: Bandwidth vs Latency

Interactive 2D ring all-reduce simulator: watch gradient chunks scatter-reduce and all-gather around a flat ring of GPUs, drag to rotate the ring, tune cluster size, model size, bandwidth and per-hop latency, and read a live compute-vs-communication timeline strip chart.

Machine Learning & Neural Networks2DAdvanced60 FPS📱 Mobile-adapted⇄ 3D version

Details

CategoryMachine Learning & Neural Networks

RenderingCanvas 2D

DifficultyAdvanced

Target FPS60

ControlsSliders for GPU count, model parameter count, interconnect bandwidth, per-GPU compute throughput and per-hop network latency; a Run Training Step button that animates one scatter-reduce plus all-gather cycle and logs it to the timeline chart; a Reset Counters button; and click-drag on the ring diagram to rotate its orientation.

distributed traininggpu clusterall-reducescaling lawsfrontier modelsinterconnectnetwork latency2d visualization

Related simulations

Frontier Models & Scaling Laws: Chinchilla Compute-Optimal ExplorerExplore the Chinchilla scaling law in 3D: watch loss L(N,D) form a surface over model size…

GRU Cell 2D: Gate Traces & Unit HeatmapA native 2D Gated Recurrent Unit simulator: watch reset and update gates, the candidate…

RC Ladder Interconnect Delay (2D)Interactive 2D RC-ladder network: watch a step signal crawl node-by-node down a chain of… ⊞ All 226 in Machine Learning & Neural Networks →

The 2D companion to the 3D ring all-reduce simulator renders the same GPU cluster gradient-sync mechanism as a flat, draggable ring diagram paired with a live compute-vs-communication timeline strip chart. Tune cluster size, model parameter count, interconnect bandwidth, per-GPU compute throughput and — new in this build — per-hop network latency, then run a training step to watch scatter-reduce and all-gather chunks circulate the ring while the timeline logs whether the cluster is compute-bound, latency-bound or bandwidth-bound.

⚙ Under the hood

2D ring all-reduce simulator: drag to rotate a flat GPU ring while gradient chunks scatter-reduce and all-gather around it, watch a live compute-vs-communication timeline strip chart, and tune cluster size, model size, bandwidth and per-hop latency to see when training tips from compute-bound to latency- or bandwidth-bound.

distributed traininggpu clusterall-reducescaling lawsfrontier modelsinterconnectnetwork latency2d visualization

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install