Monte Carlo Importance Sampling

The rare-event problem

Suppose you want to estimate the probability that a system fails — a bridge exceeds a stress threshold, a network drops more than a critical fraction of packets, a portfolio loses more than a extreme amount in a day. Naive Monte Carlo draws samples from the natural distribution and counts how often the rare event occurs. If the true probability is p, you need on the order of 1/p samples just to observe the event a handful of times, and the relative standard error of the estimate shrinks only as 1/sqrt(N·p). For p around one in a million, a reliable estimate can require billions of simulated trials — often simply not feasible.

Shift where you look

Importance sampling reframes the problem. Instead of drawing samples from the true density f(x) and hoping enough of them land in the rare region, you draw from a different, chosen proposal density g(x) that deliberately puts more mass where the event of interest happens. You then correct each sample's contribution by the likelihood ratio, or importance weight, so the estimator remains exactly unbiased for the original quantity even though you never sampled from the original distribution:

E_f[h(X)] = integral h(x) f(x) dx
          = integral h(x) * (f(x)/g(x)) * g(x) dx
          = E_g[ h(X) * w(X) ]         where w(x) = f(x) / g(x)

Estimator:  (1/N) * sum_i  h(x_i) * w(x_i),   x_i ~ g

Because g(x) was chosen to make the event common, most of your N samples now actually hit it, and each hit is correctly down-weighted by w(x) < 1 to compensate for having been over-sampled. The estimator is unbiased for any g whose support covers the support of f·h — the only mathematical requirement — but its variance depends enormously on how well g is chosen.

Choosing the proposal well

The theoretically optimal proposal — the one that reduces the estimator's variance to zero — is g*(x) ∝ |h(x)|·f(x), which is unusable in practice because computing its normalizing constant requires already knowing the quantity you are trying to estimate. In practice you settle for a reasonable, computable approximation: shift the mean of a Gaussian toward the failure region, use exponential tilting on the tail of a distribution, or borrow domain knowledge about roughly where the rare event tends to occur. Even an imperfect but sensible proposal typically cuts the required sample count by several orders of magnitude compared to naive Monte Carlo.

The way it backfires

A poorly chosen g is not merely less helpful than naive Monte Carlo — it can be actively worse. If g places too little probability mass in a region where h(x)·f(x) is non-negligible, any sample that does land there receives a huge weight w(x) = f(x)/g(x), and a handful of huge-weight samples can dominate the whole estimate, inflating its variance dramatically, in pathological cases to infinity. The practical warning sign is a small number of samples contributing a disproportionate share of the total weight — a diagnostic worth checking any time importance sampling is used for something that matters.

Where it is used

Importance sampling underlies rare-event estimation in structural reliability engineering, tail-risk estimation in quantitative finance (value-at-risk, expected shortfall), particle physics event generators, and rendering algorithms in computer graphics, where light paths that actually reach the camera are rare among all possible paths a photon could take. In every case the core idea is identical: sample where the answer lives, and pay for the shortcut with a correcting weight.

Frequently asked questions

Try it live

Everything above runs in your browser — open Monte Carlo Importance Sampling and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.