Each government alert must go out on exactly one channel, and the AI does not know in advance which channel citizens actually open. This is the classic multi-armed bandit problem: every message is a trial that both earns a reward (opened / not opened) and reveals information about that channel's true, hidden open-rate pi.
Epsilon-greedy:
with probability ε → pick a uniformly random channel (explore)
with probability 1-ε → pick argmax_i p̂_i (exploit best-so-far)
UCB1 (Upper Confidence Bound):
score_i(t) = p̂_i + sqrt( 2·ln(t) / n_i )
pick argmax_i score_i(t) (unpulled arms score +∞ and go first)
Regret after T messages:
R(T) = T·p* − Σ p_chosen(t) where p* = max_i p_i
- p̂_i is the running sample mean of opens/sends for channel i — the AI's current belief, shown as each bar's height in the top panel.
- The translucent cap above each UCB1 bar is its confidence bonus
sqrt(2·ln t / n_i) — it shrinks as a channel is sampled more, which is what makes UCB1 stop exploring a channel it has already learned about.
- Epsilon-greedy explores at a constant rate forever, so it keeps sampling weak channels occasionally even late in the run.
- Cumulative regret (bottom panel) is the gap between what an oracle (always picking the true-best channel) would have earned and what the algorithm actually earned — a flattening curve means the algorithm has converged.
Real-world relevance: this exact reward-per-arm setup drives production notification routing at scale — deciding, per user segment, whether SMS, email, push or postal mail gets the next message, and adapting as citizens' real response habits emerge.