Bayesian Updating

The Beta distribution is defined on the interval [0, 1], exactly the range of a probability, and its two shape parameters alpha and beta can be interpreted as pseudocounts of prior heads and tails. Most importantly, it is the conjugate prior for the Binomial likelihood: multiplying a Beta prior by a Binomial likelihood yields another Beta distribution. This conjugacy means the posterior update requires only two additions (alpha += k, beta += n - k) rather than a numerical integral, making computation exact and instant regardless of how many flips have been observed.

Spam filters use Bayesian updating to estimate the probability that a word (such as "free" or "winner") indicates spam, refining estimates as the user marks emails. Medical testing uses it to compute positive predictive value — the probability a positive test result means disease — by combining test sensitivity with the disease's prevalence prior. In GPS receivers, a Kalman filter applies sequential Bayesian updates to fuse noisy satellite signals with motion-model predictions. A/B testing platforms use Bayesian methods to estimate conversion rate posteriors and report credible intervals rather than p-values.

A common misconception is that Bayesian methods are "subjective" and therefore less rigorous than frequentist methods. In practice, both approaches require modeling choices — frequentists choose a test statistic and significance level, Bayesians choose a prior. When the prior is weakly informative or there is abundant data, the posterior is dominated by the likelihood and converges to the same answer a frequentist would obtain. The real difference is interpretational: Bayesians assign probabilities to hypotheses, which aligns naturally with decision-making; frequentists define probability as long-run frequency, which avoids prior specification but makes direct probability statements about parameters impossible.

The Reverend Thomas Bayes (c. 1701-1761) derived the core theorem but never published it; his friend Richard Price found the manuscript and submitted it to the Royal Society, where it appeared in Philosophical Transactions in 1763, two years after Bayes' death. Pierre-Simon Laplace independently rediscovered and greatly extended the approach around 1774, giving it the mathematical form we use today and applying it to problems ranging from birth rates to celestial mechanics. The modern term "Bayesian statistics" only became common in the 1950s-1960s, championed by statisticians such as Harold Jeffreys, Leonard Savage, and Dennis Lindley.

Bayesian Inference (Prior, Likelihood and Posterior) on this site extends the same framework to non-conjugate models. The Normal Distribution simulator demonstrates the Central Limit Theorem, which explains why posteriors from large samples become approximately Gaussian regardless of the prior. The Monte Carlo Pi simulator illustrates another probabilistic reasoning method — using random sampling to estimate quantities rather than closed-form posteriors. The Probability Distributions Explorer lets you inspect the Beta, Binomial, and other families that underpin conjugate models.

Naive Bayes classifiers apply Bayes' theorem to text and document classification, treating each word as independent evidence. Bayesian neural networks place distributions over network weights rather than point estimates, enabling uncertainty quantification in predictions. Gaussian Processes — used in Bayesian optimization for hyperparameter tuning — maintain a posterior distribution over entire functions and update it with each new experiment. Reinforcement learning algorithms such as Thompson Sampling use posterior samples to balance exploration and exploitation in multi-armed bandit problems, outperforming deterministic epsilon-greedy strategies on problems with limited data.

Scalable approximate inference is a major active area: methods such as Variational Inference (optimizing a simpler distribution to approximate the posterior) and Markov Chain Monte Carlo variants like No-U-Turn Sampler (NUTS, used in Stan and PyMC) make Bayesian updating tractable for models with millions of parameters. Bayesian deep learning research seeks to give neural networks calibrated uncertainty estimates without prohibitive computational cost. Another frontier is prior elicitation — systematically translating domain expert knowledge into prior distributions — and prior predictive checking, which evaluates whether a chosen prior generates plausible data before any observations are collected.