A real "universal" model like a sparse Mixture-of-Experts transformer (e.g. Mixtral, Switch Transformer) doesn't run one giant dense network on every input. Instead a small router scores a bank of specialist sub-networks and activates only a few of them per token:
score_i = affinity(expert_i, input) + noise
g_i = softmax(score_i / T) over all N experts
top-k = the k experts with the largest g_i
g'_i = g_i / Σ(g_j for j in top-k) (renormalized)
output = Σ g'_i · Expert_i(input) for i in top-k
- Domain buttons — pick what kind of token is arriving; each of the 6 experts has a fixed affinity profile, highest for its own specialty but with realistic overlap (e.g. the Math expert is also fairly good at Code and Logic).
- Top-k — how many experts get to process each token. Real deployed MoE models typically use k=2 out of 8–64 experts, which is why they can have huge total parameter counts while the compute cost per token stays close to a much smaller dense model.
- Temperature (T) — lowering T sharpens the softmax toward a single dominant expert (peaked routing); raising T flattens it, spreading weight across more experts even when one is clearly the best fit.
- Router entropy — Shannon entropy of the full gate distribution, in bits. Low entropy means the router is confident and specialized; high entropy means it is routing almost uniformly, which in production is usually penalized by an auxiliary load-balancing loss to keep every expert utilized.