Anomaly Detection Techniques

Anomaly Detection Techniques (Verbose)

Core Patterns

Naming the Anomaly Before Choosing a Method

"Anomaly" covers three distinct problems, and a detector built for one is close to useless on the others.

Most production incidents are contextual or collective. Teams reach for a point detector, get poor recall, and conclude anomaly detection does not work — when the actual failure was scoring raw values instead of residuals against an expectation.

Robust Univariate Screening

import numpy as np

def robust_z(x: np.ndarray) -> np.ndarray:
    med = np.median(x)
    mad = np.median(np.abs(x - med))
    if mad == 0:                              # >50% identical values
        mad = np.mean(np.abs(x - med)) or 1e-9
    return 0.6745 * (x - med) / mad

flags = np.abs(robust_z(x)) > 3.5

The classic 3-sigma rule fails through masking: the outliers you are hunting sit inside the sample used to compute σ, so a few extreme points inflate σ enough to bring themselves back inside the threshold. Ten large outliers in a thousand points can double σ and hide each other. Median and MAD have a 50% breakdown point — half the data can be arbitrary before the estimate moves.

The 0.6745 constant is Φ⁻¹(0.75), which makes MAD a consistent estimator of σ under normality, so the familiar 3-ish cutoff still means roughly what you expect. The conventional threshold is 3.5.

Heavy-tailed data (latency, revenue, file sizes) needs a log transform first, or even the robust threshold will flag the natural right tail every time.

Multivariate Detection

from sklearn.ensemble import IsolationForest
from sklearn.neighbors import LocalOutlierFactor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

iso = make_pipeline(
    StandardScaler(),
    IsolationForest(n_estimators=300, max_samples=256,
                    contamination="auto", random_state=0),
)
iso.fit(X_clean)                                  # fit on a known-good window
scores = -iso[-1].score_samples(iso[:-1].transform(X_new))   # higher = anomalous

lof = LocalOutlierFactor(n_neighbors=20, novelty=True).fit(X_clean)
lof_scores = -lof.score_samples(X_new)

Isolation Forest builds random trees and scores by average path length — anomalies sit in sparse regions and get isolated in few splits. max_samples=256 is the paper's default and is deliberately small; subsampling reduces swamping, where dense normal clusters make genuinely isolated points look ordinary.

Local Outlier Factor compares a point's local density to its neighbors'. This is the one that catches an anomaly sitting between two dense clusters — globally unremarkable, locally in a void. It requires novelty=True at construction to be usable on unseen data; the default object only exposes fit_predict on the training set.

Scaling matters more than method choice for distance- and density-based detectors: an unscaled feature measured in bytes dominates every distance computation. Fit the scaler on clean data only — one fitted on contaminated data has its variance inflated by the anomalies, which shrinks them toward the center.

Time Series Anomalies

from statsmodels.tsa.seasonal import STL

stl = STL(series, period=24 * 7, robust=True).fit()
resid = stl.resid
score = np.abs(robust_z(resid.values))

Score the residual, never the level. robust=True uses iteratively reweighted fitting so a spike does not get absorbed into the seasonal or trend component — without it, a large anomaly partly explains itself away and its residual shrinks.

For a forecast-based detector the prediction interval is the threshold: flag observations outside it, and let the interval widen where the model is genuinely uncertain instead of using one global cutoff. Multiple seasonalities (daily and weekly) need MSTL or explicit Fourier terms; a single period leaves the other cycle in the residual and produces a recurring wave of false positives.

See time-series-preprocessing for the causality rules — a detector whose expected value was fitted using future points looks excellent offline and detects nothing live.

Evaluation

from sklearn.metrics import average_precision_score, precision_recall_curve

ap = average_precision_score(y_true, scores)          # PR-AUC

order = np.argsort(-scores)
for k in (10, 50, 100, 500):
    print(k, y_true[order[:k]].mean())                # precision@k

ROC-AUC is the trap. Its x-axis is false positive rate, normalized by the huge negative class, so 10,000 false positives out of 10 million negatives moves it by 0.001 — while making the alert queue 10,000 items long. PR-AUC and precision@k use raw false positive counts and reflect what the on-call engineer actually sees.

Set k from the triage budget: if the team can investigate 40 alerts a day, the only question is how many of the top 40 are real. For streaming systems, detection latency belongs beside precision — a detector that finds an outage with 100% precision six hours later has not detected it.

Thresholding and Operations

contamination in scikit-learn sets offset_ so that the given fraction of training data is labeled anomalous. It does not affect the score ranking — so persist the raw scores and threshold them yourself against a labeled or operator-reviewed sample. Setting contamination blindly to 0.01 on data whose true rate is 0.0001 guarantees a 100:1 false positive ratio.

Feed dismissals back. A detector without a feedback loop degrades silently: data drifts, the score distribution shifts, and the threshold calibrated in March is mute or screaming by September. Recalibrate on a rolling quantile of recent scores, and exclude confirmed anomalies from the refit window.

Common Anti-Patterns

❌ Reporting accuracy. At a 0.1% base rate, a detector that does nothing scores 99.9%. ✅ Precision@k and PR-AUC, with k set by triage capacity.

❌ Reporting ROC-AUC as the headline. Stays near 0.95 while nearly every alert is false. ✅ Average precision, plus the raw confusion counts at the operating threshold.

❌ Mean ± 3σ on skewed or contaminated data. The outliers inflate σ and mask themselves. ✅ Median/MAD, and log-transform heavy-tailed quantities first.

❌ Scoring raw values on a seasonal series. Every quiet night is an anomaly. ✅ Decompose or forecast, then score the residual.

❌ Refitting nightly on data containing yesterday's undetected anomalies. The model learns them as normal and stops flagging them. ✅ Fit on a curated clean window; exclude confirmed incidents from refits.

❌ Fitting the scaler on contaminated data. Anomalies inflate the variance and get pulled toward the center. ✅ Fit scaler and detector together on the clean reference period.

❌ Trusting contamination to set the threshold. It is a guess about the base rate, not a calibration. ✅ Keep raw scores; calibrate the cutoff against reviewed samples and alert budget.

❌ A fixed threshold that never moves. Drift makes it mute or deafening. ✅ Rolling recalibration with alert-rate monitoring.

❌ No feedback capture. You never learn precision, so you cannot improve it. ✅ Record every alert's disposition; it becomes the labeled set you lacked.

Anomaly Detection Checklist

• Anomaly type identified: point, contextual, or collective
• Seasonality and trend removed before scoring time series
• Robust statistics (median/MAD) used instead of mean/σ
• Heavy-tailed features log-transformed before thresholding
• Features scaled, with the scaler fitted on clean data only
• Multivariate method used when anomalies are combinations, not extremes
• Precision@k and PR-AUC reported; accuracy and ROC-AUC not headlined
• k set from actual triage capacity
• Detection latency measured for streaming detectors
• Threshold calibrated against reviewed samples, not contamination
• Confirmed anomalies excluded from the refit window
• Alert disposition captured as feedback and as future labels
• Score distribution monitored for drift