Resampling Methods: Bootstrap & Jackknife

1. Why resample? The plug-in principle

Classical statistics gives closed-form standard errors for simple statistics like the mean (σ/√n), but for the median, trimmed mean, correlation coefficient, or a ratio of two statistics, no such formula generally exists — or it depends on unknown properties of the population distribution.

Bradley Efron's plug-in principle (1979) resolves this: since we don't know the true population distribution F, we plug in the empirical distribution F̂ — the sample itself, treating each observed value as equally likely — and estimate the sampling variability of a statistic by repeatedly resampling from F̂ instead of F.

2. The bootstrap algorithm

Given an original sample x₁, …, xₙ and a statistic θ̂ = T(x₁,…,xₙ) (e.g. the mean, median, or standard deviation), the nonparametric bootstrap works as follows:

1. Draw a bootstrap sample x₁*, …, xₙ* of size n by sampling with replacement from the original data.
2. Compute the statistic on the resample: θ̂* = T(x₁*, …, xₙ*).
3. Repeat steps 1–2 B times (typically B = 1 000–10 000) to get a bootstrap distribution θ̂*₁, …, θ̂*ᴮ.
4. Use the spread of the bootstrap distribution as an estimate of the sampling distribution of θ̂.

3. Bootstrap confidence intervals

The simplest and most widely used interval is the percentile method: sort the B bootstrap replicates and take the empirical 2.5th and 97.5th percentiles as a 95% confidence interval:

This makes no assumption of Normality and works even for skewed sampling distributions (e.g. of the median, or of a ratio). More refined variants exist for cases where the percentile method is biased:

• Basic (pivotal) bootstrap CI: reflects the percentile interval around θ̂ instead of using the replicate quantiles directly — corrects for some skew.
• BCa (bias-corrected and accelerated): adjusts both for median bias of the bootstrap distribution and for how the standard error itself changes with θ (acceleration) — the standard choice in production statistical software.

For comparison, the classical parametric CI for a mean under an assumed Normal population uses the t-distribution: X̄ ± t_n−1,0.975·s/√n. When data really are close to Normal, this and the bootstrap percentile CI nearly coincide; when data are skewed or the statistic is nonlinear (e.g. the median), they diverge and the bootstrap is more trustworthy.

4. The jackknife: leave-one-out

The jackknife (Quenouille 1949, Tukey 1958) predates the bootstrap and uses a deterministic resampling scheme: compute the statistic n times, each time leaving out exactly one observation.

Bias estimate

Variance / standard error estimate

The factor (n−1) inflates the raw variance of the leave-one-out estimates because each θ̂₍ᵢ₎ is computed from n−1 points and hence varies less than an independent replicate would — the jackknife formula corrects for this.

5. Bootstrap vs jackknife

In modern practice the bootstrap has largely superseded the jackknife for confidence intervals, but the jackknife remains useful as a cheap, deterministic bias-correction tool and underlies cross-validation and jackknife-after-bootstrap diagnostics.

6. JavaScript implementation

function mean(arr) { return arr.reduce((a,b) => a+b, 0) / arr.length; }

function median(arr) {
  const s = [...arr].sort((a,b) => a-b);
  const mid = Math.floor(s.length / 2);
  return s.length % 2 ? s[mid] : (s[mid-1] + s[mid]) / 2;
}

// ── Bootstrap: resample with replacement, B times ──────────────
function bootstrap(data, statFn, B = 2000) {
  const n = data.length;
  const replicates = new Array(B);
  for (let b = 0; b < B; b++) {
    const sample = new Array(n);
    for (let i = 0; i < n; i++) sample[i] = data[Math.floor(Math.random() * n)];
    replicates[b] = statFn(sample);
  }
  return replicates;
}

// ── Percentile confidence interval from bootstrap replicates ───
function percentileCI(replicates, alpha = 0.05) {
  const sorted = [...replicates].sort((a,b) => a-b);
  const lo = sorted[Math.floor((alpha/2) * sorted.length)];
  const hi = sorted[Math.ceil((1 - alpha/2) * sorted.length) - 1];
  return [lo, hi];
}

// ── Jackknife: leave-one-out bias and variance ─────────────────
function jackknife(data, statFn) {
  const n = data.length;
  const full = statFn(data);
  const looEstimates = new Array(n);
  for (let i = 0; i < n; i++) {
    const leaveOneOut = data.slice(0, i).concat(data.slice(i + 1));
    looEstimates[i] = statFn(leaveOneOut);
  }
  const jackMean = mean(looEstimates);
  const bias = (n - 1) * (jackMean - full);
  const variance = ((n - 1) / n) *
    looEstimates.reduce((s, v) => s + (v - jackMean) ** 2, 0);
  return { estimate: full, bias, se: Math.sqrt(variance) };
}

// ── Example: bootstrap and jackknife for the median ────────────
const data = [12, 15, 14, 10, 50, 13, 11, 16, 9, 14]; // right-skewed by outlier 50
const replicates = bootstrap(data, median, 5000);
console.log('bootstrap SE', Math.sqrt(bootstrap(data, median, 5000).reduce(
  (s,x,_,a) => s + (x - mean(a))**2, 0) / (replicates.length - 1)));
console.log('95% percentile CI', percentileCI(replicates));
console.log('jackknife (mean, not median!)', jackknife(data, mean));

7. Pitfalls

• Small samples: with n < 10–15, both methods can be unreliable — the bootstrap resample can only ever contain values already present in the original data, so extreme percentiles are poorly estimated.
• Dependent / time-series data: naive bootstrap resampling assumes i.i.d. observations. For correlated data (time series, spatial data), use the block bootstrap, which resamples contiguous blocks of observations to preserve local dependence structure.
• Statistics with unbounded influence: if a single extreme value can dominate the statistic (e.g. the maximum), the bootstrap distribution can be badly skewed or discrete — visualise the bootstrap histogram before trusting the interval.
• Computational cost at scale: for large n or expensive statistics (e.g. refitting a complex model B times), consider the faster jackknife-plus or subsampling variants instead of a full B = 10 000 bootstrap.