A mixture-of-experts (MoE) layer replaces one big feed-forward block with N smaller "expert" networks plus a gate. For an input token x with gate weight vector wi and bias bi per expert i:
score_i = w_i · x + σ·N(0,1) (noisy top-k gating, Shazeer et al. 2017)
S = top-k indices of (score_i + b_i)
g_i = softmax_{i∈S}(score_i) for i∈S, else 0
output = Σ_{i∈S} g_i · Expert_i(x)
Only the top-k experts run per token — for k≪N this makes the model's active compute far smaller than its total parameter count, which is why MoE lets large language models scale to huge parameter counts without a proportional rise in FLOPs per token.
The load-balancing problem: left alone, gradient descent tends to funnel most tokens into a handful of "winner" experts, starving the rest — those experts undertrain and the model's effective capacity collapses. This sim's bias toggle implements the auxiliary-loss-free correction popularized by DeepSeek-V3 (2024):
every batch: b_i ← b_i + γ · sign(target_load − actual_load_i)
Overloaded experts get their bias nudged down (less attractive to the router), underloaded ones get nudged up — purely through the routing decision, with no separate loss term needed. The aux loss (est.) readout shows the classic Switch-Transformer proxy N·ΣfiPi (fraction routed × mean gate probability, minimized at 1.0 when load is perfectly even across N experts).
- top-k — how many experts each token activates; higher k spreads load more evenly but costs more compute per token.
- noise σ — random jitter added to gate scores; too little and routing locks onto the same experts every time, too much and it becomes uniformly random.
- Load balancing — toggles the bias-correction rule above; watch the utilization bars flatten out once it's on.