Anomaly Detection — Isolation Forest & Autoencoders

What is Anomaly Detection?

Anomaly detection (outlier detection) is the task of identifying data points that deviate markedly from the expected pattern. The anomaly is often rare, high-impact, and hard to define explicitly: fraud transactions, manufacturing defects, network intrusions, or failing sensors.

Unlike supervised classification, anomaly detection is usually trained on mostly normal data and must recognize deviations it has never seen before. The core assumption is that normal observations are concentrated in dense regions, while anomalies live in sparse regions of the feature space.

Problem Framing & Evaluation

Before picking a model, decide how you measure success. Anomaly detection usually has a severe label imbalance and asymmetric costs.

Common evaluation metrics

• Precision at top-k — of the points flagged as most anomalous, how many are real anomalies? Useful when review bandwidth is limited.
• ROC-AUC — area under the receiver operating curve. Good for balanced-ish datasets but can be misleadingly high when the anomaly rate is tiny.
• Average Precision (PR-AUC) — preferred under extreme imbalance; focuses on the ranking of true anomalies.
• F1 at a chosen threshold — balances precision and recall after a threshold is chosen.
• Mean Average Precision (mAP) — averaged AP across multiple anomaly types if labels are multi-class.

Where P_r and R_r are precision and recall at each ranking threshold. The score rewards models that place true anomalies early in the ranked list.

How to split data cleanly

• Never put anomalies in the training set for unsupervised methods unless you are doing supervised anomaly detection.
• Reserve a contamination-free validation set of normal data to choose thresholds and model hyperparameters.
• Keep an independent test set with labeled anomalies for final evaluation.
• For temporal data, split by time so you test on future observations, not random rows.

Isolation Forest

Isolation Forest is a tree-based algorithm designed specifically for anomaly detection. Its key insight: anomalies are easier to isolate than normal points. A random tree recursively splits data on random features and thresholds; an anomalous point typically lands in a very shallow leaf, while normal points require many splits to separate.

How it works

The algorithm builds an ensemble of n_estimators isolation trees. For each tree, a random subset of the data is sampled and recursively split until every point is alone or a maximum depth is reached. The depth of a point in the tree is its path length; the average path length across trees is the anomaly score.

Here E(h(x)) is the average path length for point x, and c(n) is the average path length in a binary search tree with n samples. A score near 1 means anomaly; near 0.5 means normal; below 0.5 means very common.

Key hyperparameters

• n_estimators — number of trees in the forest. Typical: 100–500. More trees smooth the score but increase runtime.
• max_samples — number of samples used to build each tree. Defaults to 256; often sufficient because isolation trees are subsampled deliberately.
• contamination — expected fraction of anomalies in the data. Used only to set the decision threshold; if unknown, leave as "auto" and tune manually.
• max_features — number of features to consider at each split. Default 1.0 (all features); lowering it adds randomness.
• random_state — for reproducibility.

Isolation Forest in practice

from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import average_precision_score, roc_auc_score

# Train on normal data only; keep labels for evaluation
scaler = StandardScaler()
X_train_norm = scaler.fit_transform(X_train[y_train == 0])
X_test = scaler.transform(X_test)

model = IsolationForest(
    n_estimators=200,
    max_samples=256,
    contamination="auto",
    random_state=42,
    n_jobs=-1,
)
model.fit(X_train_norm)

# sklearn returns -1 for anomaly, 1 for normal; scores are negative of anomaly score
y_scores = -model.score_samples(X_test)        # higher = more anomalous
y_pred = model.predict(X_test)
y_pred = (y_pred == -1).astype(int)

print("ROC-AUC:", roc_auc_score(y_test, y_scores))
print("PR-AUC :", average_precision_score(y_test, y_scores))

When Isolation Forest shines

• Tabular data with a mix of continuous and discrete features.
• When you need fast training and inference with little tuning.
• When the anomaly mechanism is local or global deviation, not subtle semantic shifts.
• As a strong baseline before trying deep learning.

Limitations to watch

• High-dimensional data — distance and isolation become less meaningful in high dimensions; consider dimensionality reduction or deep models first.
• Clustered anomalies — if many anomalies share a pattern, Isolation Forest may treat them as a normal cluster.
• Feature scale — random splits are affected by feature range; scale features before fitting.
• Local density — it does not explicitly model local density; DBSCAN or LOF may be better for localized anomalies.

Autoencoders for Anomaly Detection

An autoencoder is a neural network trained to reconstruct its input. It consists of an encoder that compresses the input into a lower-dimensional latent representation and a decoder that expands it back. Because the model is trained primarily on normal data, it learns to reconstruct normal patterns well; unusual patterns are reconstructed poorly, producing high reconstruction error.

Architecture choices

• Fully-connected autoencoder — good for tabular data and small images. Encoder and decoder are symmetric MLPs.
• Convolutional autoencoder — best for images, video frames, or spectrograms. Uses conv layers in the encoder and transposed conv or upsampling in the decoder.
• Recurrent / LSTM autoencoder — for sequences and time series. Encodes a sequence into a fixed vector and reconstructs the sequence.
• Variational Autoencoder (VAE) — adds a probabilistic latent space. Useful when you want to model uncertainty or generate normal samples for contrast.

Training protocol

• Train only on normal data. Anomalies should be excluded or heavily down-weighted.
• Use reconstruction loss — MSE for continuous inputs, cross-entropy for binary inputs, or domain-specific losses.
• Validate on held-out normal data to detect overfitting and choose capacity.
• Score by reconstruction error on the test set; higher error = more anomalous.

PyTorch autoencoder example

import torch
import torch.nn as nn
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import average_precision_score

class Autoencoder(nn.Module):
    def __init__(self, input_dim, latent_dim=8):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Linear(32, latent_dim),
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 32),
            nn.ReLU(),
            nn.Linear(32, 64),
            nn.ReLU(),
            nn.Linear(64, input_dim),
        )

    def forward(self, x):
        return self.decoder(self.encoder(x))

# Scale and tensorize
scaler = StandardScaler()
X_train_norm = scaler.fit_transform(X_train[y_train == 0])
X_test = scaler.transform(X_test)

train_ds = torch.utils.data.TensorDataset(
    torch.tensor(X_train_norm, dtype=torch.float32),
    torch.tensor(X_train_norm, dtype=torch.float32),
)
loader = torch.utils.data.DataLoader(train_ds, batch_size=64, shuffle=True)

model = Autoencoder(input_dim=X_train_norm.shape[1])
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()

for epoch in range(50):
    for xb, _ in loader:
        optimizer.zero_grad()
        recon = model(xb)
        loss = criterion(recon, xb)
        loss.backward()
        optimizer.step()

# Score on test set
model.eval()
with torch.no_grad():
    X_test_t = torch.tensor(X_test, dtype=torch.float32)
    recon = model(X_test_t)
    errors = torch.mean((X_test_t - recon) ** 2, dim=1).numpy()

print("PR-AUC:", average_precision_score(y_test, errors))

Latent-space anomaly detection

Reconstruction error is not the only signal. You can also use the latent representation:

• Distance in latent space — fit a Gaussian or kernel density on latent vectors of normal data and flag points with low density.
• Latent reconstruction error — measure how far the latent code of a test point is from typical normal codes.
• Combined score — linearly combine reconstruction error and latent density; often more robust than either alone.

from sklearn.neighbors import KernelDensity

model.eval()
with torch.no_grad():
    z_train = model.encoder(torch.tensor(X_train_norm, dtype=torch.float32)).numpy()
    z_test = model.encoder(torch.tensor(X_test, dtype=torch.float32)).numpy()

kde = KernelDensity(bandwidth=0.5, kernel="gaussian")
kde.fit(z_train)
latent_score = -kde.score_samples(z_test)        # higher = more anomalous

combined = 0.5 * errors + 0.5 * (latent_score / latent_score.max())
print("Combined PR-AUC:", average_precision_score(y_test, combined))

Isolation Forest vs Autoencoder

Both models detect anomalies but they exploit different properties of the data.

• Use Isolation Forest when the data is tabular, features are interpretable, anomalies are global or sparse, and you need fast, low-maintenance results.
• Use an autoencoder when the data is high-dimensional, unstructured (images, sequences, embeddings), or the normal pattern is complex and non-linear.
• Combine them — run Isolation Forest on the latent code of an autoencoder, or stack both scores in an ensemble for better ranking.

Hybrid ensemble example

from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import MinMaxScaler
import numpy as np

# Fit autoencoder on normal data (as above)
model.eval()
with torch.no_grad():
    z_train = model.encoder(torch.tensor(X_train_norm, dtype=torch.float32)).numpy()
    z_test = model.encoder(torch.tensor(X_test, dtype=torch.float32)).numpy()

# Run Isolation Forest in the latent space
iso = IsolationForest(n_estimators=200, random_state=42)
iso.fit(z_train)
iso_score = -iso.score_samples(z_test)

# Combine with reconstruction error
scaler = MinMaxScaler()
scores = np.column_stack([scaler.fit_transform(errors.reshape(-1, 1)).ravel(),
                          scaler.fit_transform(iso_score.reshape(-1, 1)).ravel()])
ensemble_score = scores.mean(axis=1)
print("Ensemble PR-AUC:", average_precision_score(y_test, ensemble_score))

Practical Tips for Production

• Start simple — Isolation Forest is fast to implement and gives a strong baseline in minutes.
• Feature engineering matters — ratios, rolling statistics, and domain-specific aggregates often beat a fancier model.
• Watch for label leakage — if your features include outcomes or future information, the anomaly detector will appear perfect but fail in production.
• Monitor score drift — track the distribution of anomaly scores over time. A sudden shift may indicate data drift or a new attack pattern.
• Human-in-the-loop feedback — collect analyst labels on flagged cases and retrain or refine thresholds periodically.
• Explainability — for each flagged case, report top contributing features (per-feature reconstruction error or Isolation Forest path-length contributions).

Summary

Isolation Forest isolates anomalies through random recursive splits, making it fast and effective for tabular outliers. Autoencoders learn a compressed representation of normal data and flag points with high reconstruction error, excelling on high-dimensional and sequential data. Choose based on your feature structure, data size, and latency constraints — and always evaluate with PR-AUC or precision-at-k rather than raw accuracy.