Mixture of Experts: Sparse Gated Routing

Every glowing particle in this scene is a token leaving a gating network at the center and heading toward the small set of "expert" sub-networks it was routed to — exactly the sparse-activation mechanism (noisy top-k gating) that lets modern large language models pack far more parameters into a model than they ever touch per token. Tune top-k and the gating noise to see routing sharpen or blur, and flip on the auxiliary-loss-free load-balancing bias — the same trick used in DeepSeek-V3 — to watch a handful of overloaded experts get throttled back until utilization evens out across all six.