Bayesian Inference Explained
Bayes' theorem turns prior beliefs into posterior beliefs by incorporating evidence through the likelihood. Learn how conjugacy makes updates simple and when computational methods are needed.
📚 Fundamentals
Bayes' theorem: posterior ∝ likelihood × prior. The normalizing constant ensures probabilities integrate to one.
Conjugate Priors
- Bernoulli/Binomial: Beta prior
- Poisson: Gamma prior
- Gaussian mean (known variance): Gaussian prior
🧪 Worked Examples
Beta–Binomial Update
Prior Beta(α,β); observe s successes, f failures → Posterior Beta(α+s, β+f).
Gamma–Poisson Update
Prior Gamma(k, θ); observe count c over exposure t → Posterior Gamma(k+c, θ/(1+θt)).
🚀 Advanced Concepts
MCMC
When conjugacy is unavailable, Markov chain Monte Carlo approximates posteriors by sampling.
Variational Inference
Optimize a simpler family to approximate a complex posterior with an evidence lower bound.
❓ Frequently Asked Questions
1) What is a prior?
A probability distribution encoding beliefs before seeing data.
A probability distribution encoding beliefs before seeing data.
2) How to choose a prior?
Use domain knowledge, reference priors, or weakly informative defaults.
Use domain knowledge, reference priors, or weakly informative defaults.
3) What is conjugacy?
A prior family that yields a posterior of the same family for easy updates.
A prior family that yields a posterior of the same family for easy updates.
4) Does Bayes need large samples?
No; it works for any sample size, with priors moderating small-sample noise.
No; it works for any sample size, with priors moderating small-sample noise.
5) Credible vs confidence intervals?
Credible intervals are probability statements about parameters.
Credible intervals are probability statements about parameters.
6) How sensitive to priors?
Do sensitivity analysis; the effect diminishes with more data.
Do sensitivity analysis; the effect diminishes with more data.
7) MAP vs posterior mean?
MAP maximizes posterior density; mean minimizes squared error risk.
MAP maximizes posterior density; mean minimizes squared error risk.
8) Hierarchical Bayes?
Priors on priors to share strength across related groups.
Priors on priors to share strength across related groups.
9) Model comparison?
Use Bayes factors or information criteria (WAIC, LOO).
Use Bayes factors or information criteria (WAIC, LOO).
10) Computational tools?
Stan, PyMC, NumPyro, Turing.jl for modern Bayesian workflows.
Stan, PyMC, NumPyro, Turing.jl for modern Bayesian workflows.