Stacking Ensembles: Meta-Learner Blending

Stacking Setup

Meta-learner training Naive (in-sample)

CV folds K 5

Surface shown

Live Metrics

Fit accuracy—

Held-out test accuracy—

Generalization gap—

Class 0 (low probability) Class 1 (high probability) Columns = predicted P(class 1) over the feature plane. Dots = training points at their true label height.

How it works

Three base learners are trained on the same 2-feature synthetic dataset and each outputs a probability P(class 1) at every point (x, y):

f1 (kernel k-NN):   weighted vote of nearby points
  w_i = exp(-d(x,p_i)^2 / (2*sigma^2))
  f1(x) = sum(w_i * label_i) / sum(w_i)

f2 (logistic regr): f2(x,y) = sigmoid(a*x + b*y + c)
  fit by gradient descent on cross-entropy loss

f3 (decision stump): f3 = sigmoid(steep * signed distance
  to the single threshold that best separates the classes
  on one axis, found by brute-force search)

The stacking meta-learner is a second logistic regression trained on the three base outputs themselves:

P_final(x,y) = sigmoid(w1*f1 + w2*f2 + w3*f3 + b)

Naive mode trains the meta-learner on base predictions computed by models that already saw those exact training points — the tight-kernel k-NN in particular can nearly memorize its own training set (self-distance = 0 dominates the kernel sum), so the meta-learner learns to trust a leaked, over-confident signal.
Cross-validated mode splits the training set into K folds; each point's base features come from models trained only on the other K−1 folds, so the meta-learner only ever sees honest, out-of-fold predictions — exactly how real stacked generalization (Wolpert, 1992) avoids leakage.
Both modes evaluate on the same unseen held-out test set with the same base learners, so the "held-out test accuracy" is a fair comparison — watch the generalization gap (fit − test accuracy) shrink when you switch to cross-validated stacking.

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.