Thompson Sampling: Bayesian Multi-Armed Bandit

Bandit Setup

Number of arms 4

Autoplay speed 3 rounds/s

Reveal true win probabilities

Live Stats

Round

Reward pulled

Cumulative regret

0.00

Sampled arm each round: —

Arms — Beta(α, β) posterior

How it works

Each arm's unknown win probability is tracked as a Bayesian posterior belief, a Beta(α, β) distribution starting at the uninformative Beta(1, 1). Every round the agent:

1. Sample θ_i ~ Beta(α_i, β_i)  for every arm i
2. Pull arm i* = argmax_i θ_i
3. Observe reward r ∈ {0,1} ~ Bernoulli(p_i*)
4. Update: α_i* += r,  β_i* += (1 − r)

The curve above each pedestal is the Beta pdf itself, normalised to a fixed height so you can watch its shape evolve: wide and flat at first (total uncertainty), narrowing and shifting toward the arm's true win rate as evidence accumulates. An arm that keeps disappointing gets a posterior that shrinks toward 0 and is rarely sampled highest again — that is exploration and exploitation falling out of the same sampling rule, with no separate ε or confidence-bound parameter to tune.

Cumulative regret — the gap between always pulling the (secretly) best arm and what the agent actually earned: Σ(p_best − p_chosen). A good bandit algorithm keeps this growing sub-linearly with the round count.
Reveal true probabilities — shows the hidden Bernoulli win rates so you can watch the posteriors race to find them.

Real-world relevance: this is the exact mechanism behind live A/B-style traffic allocation, ad and content ranking, and clinical-trial arm allocation — anywhere a system must keep learning which option is best while still earning reward along the way.

Round 0 — last pull: —

Posterior density Beta(α,β) This round's sampled θ

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.