Autoencoders for Anomaly Detection: Learning Normal to Spot the Abnormal
How autoencoder neural networks flag anomalies by measuring reconstruction error, and how they compare with density-based methods like DBSCAN, LOF and One-Class SVM.
The idea: compress, then judge by how badly you can rebuild
Most anomaly-detection algorithms ask "does this point look far from its neighbours?" An autoencoder asks a different question: "if I squeeze this data point down to a tiny representation and then try to expand it back out, how close do I get to the original?" A neural network built as an autoencoder has two halves — an encoder that compresses an input down to a small bottleneck layer, and a decoder that tries to reconstruct the full input from that compressed form. The network is trained only on normal data, so it becomes very good at reconstructing patterns it has seen many times before.
Once trained, the network is shown new data, including data it has never encountered. Normal examples, which resemble the training distribution, get reconstructed accurately: the output is close to the input, and the reconstruction error — typically the mean squared difference between input and output — stays low. Anomalous examples, which do not fit the patterns the network learned, get reconstructed poorly, because the bottleneck was never optimised to preserve whatever makes them unusual. A large reconstruction error is therefore treated as evidence of an anomaly.
Building the network and choosing the bottleneck
A simple autoencoder for tabular sensor data is just a stack of dense layers that narrows down to a bottleneck and then widens back out symmetrically, for example an input layer, a hidden layer of 32 units, a bottleneck of 8 units, another hidden layer of 32 units, and an output layer matching the original input dimension. The network is trained with a reconstruction loss such as mean squared error, using the Adam optimiser, exactly as a regression model would be trained — except the target is the input itself.
The bottleneck size is the key design decision. Too large, and the network has enough capacity to memorise unusual inputs as well as normal ones, which defeats the purpose — everything reconstructs well, including anomalies, so the model loses discriminating power. Too small, and even normal variation cannot be reconstructed accurately, producing a high baseline error that drowns out the anomaly signal. In practice the bottleneck is chosen empirically, often as a fraction of the number of input features, and validated by checking that reconstruction error separates known normal and anomalous examples cleanly.
Setting the anomaly threshold
Reconstruction error on its own is just a number; turning it into a "normal or anomaly" decision requires a threshold, and this is where most of the practical judgement calls live. A common approach is percentile-based: compute reconstruction error across a validation set of mostly-normal data, then flag anything above the 95th or 99th percentile as an anomaly. A second option is statistical: model the error as approximately normal and flag anything beyond the mean plus three standard deviations. A third, when a small amount of labelled anomaly data exists, is to sweep the threshold and pick the value that maximises F1-score or another chosen metric on a validation set.
None of these choices is free of trade-offs. A stricter threshold reduces false positives — engineers are not paged for normal fluctuation — but raises the risk of missing genuinely anomalous events, which matters if the anomalies represent safety issues or equipment failures. A looser threshold catches more true anomalies but risks alert fatigue, where operators start ignoring alerts because too many turn out to be benign. The threshold should ultimately be set with the downstream cost of each error type in mind, not purely by a statistical rule of thumb.
Density-based alternatives: DBSCAN and Local Outlier Factor
Autoencoders are not the only unsupervised route to anomaly detection, and simpler methods are often tried first because they need no neural network training. DBSCAN, a clustering algorithm built around local point density, naturally produces anomaly labels as a side effect: it groups points that are densely packed together into clusters, and any point that does not have enough neighbours within a given radius to join a cluster is labelled as noise. Those noise points are the anomalies. DBSCAN needs two parameters — the neighbourhood radius (eps) and the minimum number of points required to form a dense region (min_samples) — and it requires feature scaling beforehand, since it works directly on the geometry of the data and features on very different scales would distort distance calculations.
Local Outlier Factor (LOF) takes a related but more nuanced approach: rather than a hard density cutoff, it compares the local density around a point to the local density around its neighbours. A point whose neighbourhood is much sparser than its neighbours' neighbourhoods gets a high outlier score, even if the point sits in a region of the data that is moderately dense in absolute terms. This makes LOF better suited than DBSCAN to data where different regions have naturally different densities — for example, sensor readings that cluster tightly during quiet periods but spread out more during busy periods, where a fixed-radius method like DBSCAN would misjudge which points are truly anomalous.
One-Class SVM and when it earns its cost
One-Class SVM takes yet another approach: it tries to learn a boundary — in a transformed, often higher-dimensional feature space via a kernel function — that encloses the region occupied by normal data as tightly as possible, then flags anything falling outside that boundary as anomalous. It can capture non-linear, complex-shaped normal regions that a simple distance-based rule would miss, and it works reasonably well on smaller datasets. Its weaknesses are computational cost, which scales poorly as the dataset grows, and the difficulty of interpreting exactly why a boundary excluded a given point, since the boundary lives in a transformed feature space rather than the original one.
Choosing among these four methods — autoencoders, DBSCAN, LOF, and One-Class SVM — is mostly a question of data shape and scale. Autoencoders handle high-dimensional data with complex, non-linear structure well, and scale to large datasets once trained, but need enough training data and compute to learn a good bottleneck representation and offer the least interpretability of the group. DBSCAN is fast and interpretable when the data has a genuine cluster structure. LOF handles varying-density regions better than DBSCAN, at somewhat higher computational cost. One-Class SVM suits smaller datasets with a well-defined normal region but does not scale gracefully to large or very high-dimensional data.
Operating anomaly detection in production
Whichever algorithm is chosen, deploying anomaly detection is not a one-off training exercise. The definition of "normal" tends to drift as underlying systems age, usage patterns change, or seasons shift, so models need periodic retraining on recent data rather than being frozen at initial deployment. Every flagged anomaly should be logged with enough context — the raw values, the reconstruction error or density score, the timestamp — to support later review, and ideally there is a feedback loop where the humans who investigate flagged events confirm or reject them, generating a growing pool of labelled examples that can be used to validate future threshold choices or even train a supervised classifier later on.
Evaluation should go beyond a single number. Precision measures what fraction of flagged anomalies were genuine, which matters for keeping operator trust; recall measures what fraction of real anomalies were caught, which matters for safety and reliability; and when a labelled test set is available, AUC-ROC gives a threshold-independent view of how well the anomaly score separates normal from abnormal overall. Different alerting severities — a minor deviation logged quietly versus a major one triggering an immediate page — let the same underlying anomaly score drive proportionate responses instead of a single blunt on/off decision.
Frequently Asked Questions
Why would I use an autoencoder instead of Isolation Forest?
Isolation Forest is fast, needs little tuning, and works well on tabular data with moderate dimensionality. Autoencoders earn their extra training cost when the data is high-dimensional and has non-linear structure that a tree-based splitting approach struggles to isolate efficiently — for example raw sensor waveforms, images, or wide feature sets with complex correlations between variables.
Does an autoencoder need labelled anomaly data to train?
No. It only needs a training set that is predominantly normal data — it never sees examples labelled as anomalies during training. It learns to reconstruct normal patterns well, and anomalies are identified afterward purely by their high reconstruction error, which makes it a genuinely unsupervised method.
How is LOF different from DBSCAN if both are density-based?
DBSCAN uses a single global density threshold: any point without enough neighbours within a fixed radius is noise. LOF compares each point's local density to its neighbours' local density, so it can flag outliers in sparse regions even when the overall dataset has widely varying density from one region to another, something a single fixed radius cannot adapt to.
What is a reasonable reconstruction error threshold to start with?
A common starting point is the 95th or 99th percentile of reconstruction error measured on a validation set of mostly-normal data — flag anything above that as an anomaly. Treat this as a first estimate, not a final answer; adjust it based on how many false positives operators can tolerate versus how many missed anomalies would cost.
Can these unsupervised methods replace supervised classifiers entirely?
Usually not entirely. Unsupervised methods are essential when labelled anomaly examples are scarce or anomalies are too rare and varied to have been seen before. Once enough confirmed anomalies accumulate through operator feedback, many teams add a supervised classifier on top, using the unsupervised score as one input feature alongside the raw data.