Unsupervised Anomaly Detection

Unsupervised anomaly detection identifies unusual patterns without labelled examples of anomalies. This is essential when anomalies are rare, unknown, or constantly evolving. Instead of learning what anomalies look like, these methods learn what normal looks like and flag deviations.

Where It's Used

Unsupervised anomaly detection applies across domains:

• Cybersecurity: Intrusion detection, unusual network activity
• Manufacturing: Defect detection, process anomalies
• Finance: Unusual transactions, market manipulation
• Healthcare: Abnormal patient readings, equipment faults
• IT Operations: Infrastructure failures, performance degradation
• IoT: Sensor malfunctions, environmental anomalies

Products: Splunk UBA, AWS GuardDuty, Azure Sentinel, Datadog.

Why Unsupervised?

Typical Data

What You Actually Do

1. Understand the domain - What could be anomalous and why it matters
2. Collect normal data - Ideally anomaly-free training set
3. Engineer features - Domain-relevant representations
4. Handle missing values - Imputation or indicator features
5. Normalise appropriately - Robust scaling for outlier presence
6. Choose method - Isolation Forest, autoencoder, clustering
7. Set threshold - Based on contamination estimate or business needs
8. Validate with experts - Review detected anomalies
9. Deploy with monitoring - Track detection rates
10. Iterate on feedback - Incorporate analyst decisions

Baseline First: Isolation Forest

Fast, effective, and requires minimal tuning:

from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import RobustScaler
import numpy as np

class IsolationForestDetector:
    def __init__(self, contamination=0.01, n_estimators=100):
        self.scaler = RobustScaler()
        self.model = IsolationForest(
            contamination=contamination,
            n_estimators=n_estimators,
            random_state=42,
            n_jobs=-1
        )

    def fit(self, X):
        X_scaled = self.scaler.fit_transform(X)
        self.model.fit(X_scaled)
        return self

    def predict(self, X):
        X_scaled = self.scaler.transform(X)
        # Returns -1 for anomalies, 1 for normal
        predictions = self.model.predict(X_scaled)
        return predictions == -1  # True for anomalies

    def score(self, X):
        X_scaled = self.scaler.transform(X)
        # Lower score = more anomalous
        return -self.model.decision_function(X_scaled)

# Usage
detector = IsolationForestDetector(contamination=0.01)
detector.fit(X_train)
anomalies = detector.predict(X_test)
scores = detector.score(X_test)

Detection Methods

Autoencoder Detector:

import torch
import torch.nn as nn
import numpy as np

class AnomalyAutoencoder(nn.Module):
    def __init__(self, input_dim, encoding_dim=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, encoding_dim)
        )
        self.decoder = nn.Sequential(
            nn.Linear(encoding_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 128),
            nn.ReLU(),
            nn.Linear(128, input_dim)
        )

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded

class AutoencoderDetector:
    def __init__(self, input_dim, threshold_percentile=95):
        self.model = AnomalyAutoencoder(input_dim)
        self.threshold_percentile = threshold_percentile
        self.threshold = None

    def fit(self, X, epochs=50, lr=1e-3):
        X_tensor = torch.FloatTensor(X)
        optimizer = torch.optim.Adam(self.model.parameters(), lr=lr)
        criterion = nn.MSELoss()

        self.model.train()
        for epoch in range(epochs):
            optimizer.zero_grad()
            reconstructed = self.model(X_tensor)
            loss = criterion(reconstructed, X_tensor)
            loss.backward()
            optimizer.step()

        # Set threshold from training data
        self.model.eval()
        with torch.no_grad():
            recon = self.model(X_tensor)
            errors = torch.mean((X_tensor - recon) ** 2, dim=1).numpy()
        self.threshold = np.percentile(errors, self.threshold_percentile)

    def predict(self, X):
        X_tensor = torch.FloatTensor(X)
        self.model.eval()
        with torch.no_grad():
            recon = self.model(X_tensor)
            errors = torch.mean((X_tensor - recon) ** 2, dim=1).numpy()
        return errors > self.threshold

    def score(self, X):
        X_tensor = torch.FloatTensor(X)
        self.model.eval()
        with torch.no_grad():
            recon = self.model(X_tensor)
            errors = torch.mean((X_tensor - recon) ** 2, dim=1).numpy()
        return errors

Local Outlier Factor:

from sklearn.neighbors import LocalOutlierFactor

class LOFDetector:
    def __init__(self, n_neighbors=20, contamination=0.01):
        self.model = LocalOutlierFactor(
            n_neighbors=n_neighbors,
            contamination=contamination,
            novelty=True  # For prediction on new data
        )

    def fit(self, X):
        self.model.fit(X)
        return self

    def predict(self, X):
        return self.model.predict(X) == -1

    def score(self, X):
        return -self.model.decision_function(X)

Threshold Selection

import numpy as np

class ThresholdSelector:
    @staticmethod
    def percentile_threshold(scores, percentile=95):
        """Top percentile as anomalies."""
        return np.percentile(scores, percentile)

    @staticmethod
    def iqr_threshold(scores, multiplier=1.5):
        """IQR-based threshold."""
        q1, q3 = np.percentile(scores, [25, 75])
        iqr = q3 - q1
        return q3 + multiplier * iqr

    @staticmethod
    def mad_threshold(scores, multiplier=3.5):
        """Median Absolute Deviation threshold."""
        median = np.median(scores)
        mad = np.median(np.abs(scores - median))
        return median + multiplier * 1.4826 * mad

    @staticmethod
    def contamination_threshold(scores, contamination=0.01):
        """Fixed contamination rate."""
        return np.percentile(scores, 100 * (1 - contamination))

Evaluation Metrics

Common Mistakes & Checklist

• [ ] Training on contaminated data: Anomalies in training set skew normal model
• [ ] Wrong normalisation: Standard scaling affected by outliers
• [ ] Ignoring feature correlation: Some methods assume independence
• [ ] Static threshold: Doesn't adapt to distribution changes
• [ ] Over-alerting: Too many false positives causes alert fatigue

Production Checklist

Before deployment: - [ ] Validate on known anomalies if available - [ ] Set threshold for acceptable false positive rate - [ ] Build review interface for analysts - [ ] Plan for feedback incorporation

After deployment: - [ ] Monitor detection rate stability - [ ] Track analyst feedback - [ ] Retrain on updated normal data - [ ] Adjust thresholds based on feedback

Tools & Stack

Mini Case Example

Scenario: Manufacturing defect detection from sensor data.

Approach: - Isolation Forest on 50 sensor features - Autoencoder for complex pattern detection - Ensemble of both methods

Results: - 87% of defects caught (recall) - 5% false positive rate - 2 second detection latency

Related Pages

• Index: ML Field Guide
• Fraud Detection
• Time Series Anomaly Detection
• Predictive Maintenance

Further Reading

• PyOD Documentation
• Anomaly Detection Survey
• Deep Learning for Anomaly Detection

Try it live

Everything above runs in your browser — open AI Anomaly Detection Lab — A Manufacturing Sensor Case Study and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.