Bayesian Machine Learning: Turning Belief Into a Number

Three ingredients, one rule

Bayesian inference starts from a prior distribution — what you believe about a parameter before seeing data — and a likelihood — how probable the observed data is under different parameter values. Bayes' rule combines them into a posterior: updated belief, in light of evidence.

For estimating a coin's bias θ, the Beta distribution paired with the Binomial likelihood is the canonical worked example, because the posterior is always another Beta distribution: start with Beta(α, β), observe h heads and t tails, and the posterior is exactly Beta(α+h, β+t). No numerical integration needed.

Strong priors resist evidence, weak priors chase it

A Beta(20,20) prior already encodes the weight of 40 imaginary observed flips split evenly — new data barely moves it. A Beta(1,1) prior (uniform, "I have no idea") swings dramatically after just a handful of flips. Neither is wrong; they encode different starting confidence, and the posterior is honest about how much either one shifted.

Credible intervals vs. confidence intervals

A 90% credible interval is a range that the posterior distribution assigns 90% probability to containing the true parameter — a direct probability statement about the unknown quantity. A frequentist confidence interval instead makes a statement about the long-run behaviour of the estimation procedure across repeated sampling. They often look numerically similar but answer different questions.

Where this shows up in production systems

Bayesian optimization uses a probabilistic model of an unknown function to decide which hyperparameters to try next. Thompson sampling applies the same beta-binomial machinery to multi-armed bandit problems like ad selection. Bayesian neural networks put a distribution over every weight instead of a single learned value, trading extra computation for calibrated uncertainty estimates the model can report alongside its predictions.