Presets
Playback
Data set
A 3-D point cloud lying on a curved or flat surface.
Code size K
K = 1 is a line code, K = 2 a surface code, K = 3 no bottleneck.
Activation
tanh makes the code nonlinear; linear reproduces PCA.
Probe point
Move the probe: its reconstruction and error are shown, and its activations light the network.
Architecture
Training
Adam with mini-batches; denoising noise and weight decay are optional.
Camera
Layers
Quality
Model checks
The science
Model
An autoencoder is a network that is trained to output its own input through a narrow middle layer. The encoder z = f(x) maps a point x in R^D to a short code z in R^K with K < D; the decoder x^ = g(z) reconstructs it. This page uses D -> H -> K -> H -> D with tanh hidden units and linear code and output layers. The loss is the mean squared reconstruction error per sample, L = (1/N) sum |x - g(f(x))|^2 + lambda |W|^2.
Training
The gradient is computed by back-propagation (chain rule through the four layers) and the weights are updated with Adam, m = b1 m + (1 - b1) g, v = b2 v + (1 - b2) g^2, w -= lr m^ / (sqrt(v^) + eps). The page verifies the analytic gradient against central finite differences. A denoising autoencoder corrupts the input with Gaussian noise sigma but still asks for the clean point, which makes the decoded curve a smooth projection onto the data manifold.
Link to PCA
With linear units the optimal K-dimensional code spans the first K principal components (Baldi and Hornik, 1989), and the smallest possible error is the sum of the discarded covariance eigenvalues, L_min = lambda_(K+1) + ... + lambda_D. The orange line in the loss graph is this bound, computed with an independent Jacobi eigen-solver. A nonlinear network can go below it when the data lie on a curved manifold, which a straight PCA axis cannot follow.
Anomaly detection
A point far from the learned manifold is reconstructed badly. The page marks the 95th percentile of the training errors; a probe point above it is flagged as an anomaly. This is the standard way autoencoders are used to find faults in sensor data.
What is simplified
The data are small synthetic manifolds with a little Gaussian noise, the network is tiny and trained with mini-batch Adam without a validation split, and real image autoencoders use convolutional layers and far more data. The latent colours are only a visual aid. A variational autoencoder (which adds a KL term and sampling) is not modelled here.
What am I looking at
- Cloud of spheres: the training points, coloured by the latent code z of the first unit.
- Yellow curve or sheet: the decoder image g(z): everything the network can output.
- Red links: the reconstruction error of each point.
- Network sculpture: nodes are lit by the activations of the probe point; wires are orange for positive and blue for negative weights.
- Glowing sphere: the probe; green when it is normal and red when it is an anomaly.
What to try
- Pick Linear data = PCA: the loss meets the orange PCA bound, and the learned axis coincides with the dashed PCA line.
- Pick Arc with a linear autoencoder, then switch the activation to tanh: the loss drops far below the PCA bound because the decoded yellow curve now bends.
- Set K to 2 (no bottleneck): the network learns the identity and the anomaly map becomes flat, so nothing is detected.
- Drag the probe off the curve and watch it turn red once its error passes the 95th percentile.
- Raise the denoising sigma and see the reconstruction links pull noisy points back onto the manifold.