Unlike a scripted loss curve, this simulator actually trains a model: a degree-D polynomial (Chebyshev basis) is fit by real full-batch gradient descent against a small noisy synthetic dataset, one epoch at a time. Training and validation loss are both genuine mean-squared error, recomputed from the model's actual current weights.
With enough capacity (high degree) and few enough points, gradient descent exhibits implicit regularization by iteration count: early epochs capture the broad, low-order shape of the true function — which also fits the held-out validation set well — while later epochs bend the curve to chase noise specific to the training points, which the validation set does not share. That is exactly why training loss falls monotonically while validation loss falls, bottoms out, then rises: real overfitting, not a hand-drawn curve.
best = +inf, best_epoch = 0, bad = 0
for epoch e = 0, 1, 2, ...:
w -= lr * grad_MSE(w; train_batch) // real GD step
v = MSE(w; validation_set)
if v < best - min_delta:
best = v; best_epoch = e; bad = 0
else:
bad += 1
if bad >= patience:
stop training at epoch e
restore weights from best_epoch
- Patience — consecutive non-improving epochs tolerated before halting.
- Min Δ — how large a validation-loss drop must be to reset the patience counter, so measurement noise doesn't fool the algorithm.
- Model capacity (degree) — a low-degree polynomial can't represent noise even with unlimited epochs (little/no overfitting); a high-degree one has enough free parameters to eventually memorise the training points, so overfitting appears earlier and more sharply.
- Data noise σ — the actual noise standard deviation added when the synthetic dataset is generated. Zero noise means nothing to overfit to, however high the capacity.
The top plot shows the raw data (blue = train points, orange = val points) and the model's current fitted curve (green) — watch it stay smooth early on, then wobble to chase individual training dots once overfitting begins. The gold marker on the loss plot is the best checkpoint kept; the red line marks the actual halt epoch.