Probability and Statistics: From Fundamentals to Modern Applications

Probability Foundations

Sample space Ω, events, probability measure P satisfying Kolmogorov axioms. Conditional probability: P(A|B) = P(A∩B)/P(B). Bayes' theorem: P(A|B) = P(B|A)P(A)/P(B). Independence: P(A∩B) = P(A)P(B). Random variables: discrete (PMF) and continuous (PDF). Expectation: E[X] = Σ xP(X=x) or ∫xf(x)dx. Variance: Var(X) = E[(X-μ)²] = E[X²] - (E[X])². Common distributions: Bernoulli, Binomial, Poisson (rare events), Geometric (first success). Continuous: Uniform, Normal (Gaussian), Exponential, Gamma. Central Limit Theorem: sample mean of n iid observations approaches Normal(μ, σ²/n) as n→∞ — foundational for statistical inference.

Statistical Inference

Point estimation: MLE (Maximum Likelihood Estimation) — find θ maximizing L(θ|data). Properties: consistency, asymptotic normality, efficiency (Cramér-Rao bound). Method of moments: equate sample moments to population moments. Confidence intervals: 95% CI for mean: x̄ ± 1.96·σ/√n. Hypothesis testing: null H₀ vs alternative H₁. Test statistic → p-value. Type I error (α): rejecting true H₀. Type II error (β): failing to reject false H₀. Power = 1-β. Common tests: z-test, t-test (small samples, unknown σ), chi-squared (categorical), F-test (variance comparison), ANOVA (multiple groups). Multiple testing: Bonferroni correction, Benjamini-Hochberg FDR control.

Bayesian Statistics

Bayesian framework: posterior ∝ likelihood × prior. P(θ|data) ∝ P(data|θ)P(θ). Prior encodes beliefs before data. Conjugate priors: Beta-Binomial, Normal-Normal, Gamma-Poisson — closed-form posteriors. Non-conjugate: Markov Chain Monte Carlo (MCMC) sampling. Metropolis-Hastings, Gibbs sampling, Hamiltonian Monte Carlo (HMC). Stan: probabilistic programming language for Bayesian models. Bayesian advantages: natural uncertainty quantification, works with small data, incorporates domain knowledge. Hierarchical/multilevel models: parameters at multiple levels (students within schools). Model comparison: Bayes factors, WAIC, LOO-CV. Bayesian deep learning: uncertainty in neural network predictions.

Regression and Classification

Linear regression: y = Xβ + ε, OLS estimator β̂ = (XᵀX)⁻¹Xᵀy. Assumptions: linearity, independence, homoscedasticity, normality of errors. R²: proportion of variance explained. Regularization: Ridge (L2: λ‖β‖²) shrinks coefficients, Lasso (L1: λ‖β‖₁) performs feature selection. Elastic Net: combines L1 and L2. Logistic regression: P(Y=1) = σ(Xβ), binary classification via sigmoid function. Generalized Linear Models (GLM): link function g(E[Y]) = Xβ — Poisson regression (log link), gamma regression. Random forests: ensemble of decision trees with bagging and feature subsampling. Gradient boosting (XGBoost, LightGBM): sequential tree building, minimizing loss gradient.

Modern Applications

A/B testing: randomized controlled experiments in tech (sample size calculation, sequential testing). Causal inference: counterfactuals, potential outcomes (Rubin), DAGs (Pearl). Methods: difference-in-differences, regression discontinuity, instrumental variables, propensity score matching. Time series: ARIMA, GARCH (volatility), Prophet (Facebook), state-space models. Survival analysis: Kaplan-Meier curves, Cox proportional hazards (clinical trials). Spatial statistics: kriging, Gaussian processes, geospatial modeling. Bayesian optimization: hyperparameter tuning for ML models. Information theory: entropy H(X) = -Σ p(x)log p(x), cross-entropy loss in neural networks, KL divergence.

Try it live

Everything above runs in your browser — open Spirograph and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.