Autoencoder Visualization - 3D Lab

A network squeezes each point through a narrow code and tries to rebuild it. Train it and watch the decoder learn the shape of the data.

Presets

Playback

Data set

A 3-D point cloud lying on a curved or flat surface.

Code size K

K = 1 is a line code, K = 2 a surface code, K = 3 no bottleneck.

Activation

tanh makes the code nonlinear; linear reproduces PCA.

Probe point

Move the probe: its reconstruction and error are shown, and its activations light the network.

Architecture

Training

Adam with mini-batches; denoising noise and weight decay are optional.

Camera

Layers

Quality

Model checks

The science

Model

An autoencoder is a network that is trained to output its own input through a narrow middle layer. The encoder z = f(x) maps a point x in R^D to a short code z in R^K with K < D; the decoder x^ = g(z) reconstructs it. This page uses D -> H -> K -> H -> D with tanh hidden units and linear code and output layers. The loss is the mean squared reconstruction error per sample, L = (1/N) sum |x - g(f(x))|^2 + lambda |W|^2.

Training

The gradient is computed by back-propagation (chain rule through the four layers) and the weights are updated with Adam, m = b1 m + (1 - b1) g, v = b2 v + (1 - b2) g^2, w -= lr m^ / (sqrt(v^) + eps). The page verifies the analytic gradient against central finite differences. A denoising autoencoder corrupts the input with Gaussian noise sigma but still asks for the clean point, which makes the decoded curve a smooth projection onto the data manifold.

Link to PCA

With linear units the optimal K-dimensional code spans the first K principal components (Baldi and Hornik, 1989), and the smallest possible error is the sum of the discarded covariance eigenvalues, L_min = lambda_(K+1) + ... + lambda_D. The orange line in the loss graph is this bound, computed with an independent Jacobi eigen-solver. A nonlinear network can go below it when the data lie on a curved manifold, which a straight PCA axis cannot follow.

Anomaly detection

A point far from the learned manifold is reconstructed badly. The page marks the 95th percentile of the training errors; a probe point above it is flagged as an anomaly. This is the standard way autoencoders are used to find faults in sensor data.

What is simplified

The data are small synthetic manifolds with a little Gaussian noise, the network is tiny and trained with mini-batch Adam without a validation split, and real image autoencoders use convolutional layers and far more data. The latent colours are only a visual aid. A variational autoencoder (which adds a KL term and sampling) is not modelled here.

What am I looking at
  • Cloud of spheres: the training points, coloured by the latent code z of the first unit.
  • Yellow curve or sheet: the decoder image g(z): everything the network can output.
  • Red links: the reconstruction error of each point.
  • Network sculpture: nodes are lit by the activations of the probe point; wires are orange for positive and blue for negative weights.
  • Glowing sphere: the probe; green when it is normal and red when it is an anomaly.
What to try
  • Pick Linear data = PCA: the loss meets the orange PCA bound, and the learned axis coincides with the dashed PCA line.
  • Pick Arc with a linear autoencoder, then switch the activation to tanh: the loss drops far below the PCA bound because the decoded yellow curve now bends.
  • Set K to 2 (no bottleneck): the network learns the identity and the anomaly map becomes flat, so nothing is detected.
  • Drag the probe off the curve and watch it turn red once its error passes the 95th percentile.
  • Raise the denoising sigma and see the reconstruction links pull noisy points back onto the manifold.
Loading model...
Loading Blender model...
Training loss (log) and the PCA bound
Distribution of the latent code
Reconstruction error of every point
Decoder output as a function of the code
Made with 🟠 Blender · Three.js