anomaly-detection

name	anomaly-detection
description	>-

Core Principles

1. Define normal first — You cannot detect anomalies without a clear model of what "normal" looks like. Start by characterizing the baseline.
2. Context over thresholds — A value that's anomalous in one context is expected in another. Day-of-week, seasonality, and deployment events all shift what's normal.
3. False positives are expensive — An alerting system that cries wolf gets ignored. Optimize for precision (low false positives) before recall (catching everything).
4. Anomaly ≠ problem — An anomaly is a statistical observation, not an automatic incident. Always pair detection with investigation.
5. Simple methods first — Z-scores and IQR catch most anomalies. Don't reach for Isolation Forests until simple methods have failed.
6. Stationarity matters — Most statistical methods assume your data doesn't change character over time. If it does, you need methods that handle trend, seasonality, or concept drift.

---

Methodology Reference

Tier 1: Statistical Methods (Start Here)

Use these when data is well-understood, roughly stationary, and you can define "normal" numerically.

Method	How It Works	Best For	Assumptions	Limitations
Z-Score	Flag points > k standard deviations from the mean	Univariate, roughly normal data	Normal distribution, no trend	Fails with skewed data, sensitive to outliers in training set
Modified Z-Score (MAD)	Uses median absolute deviation instead of std dev	Same as Z-score but robust to existing outliers	Symmetric distribution	Less sensitive than Z-score for normally distributed data
IQR (Interquartile Range)	Flag points outside Q1 − 1.5·IQR or Q3 + 1.5·IQR	Non-normal, skewed distributions	None (non-parametric)	Misses anomalies in heavy-tailed distributions
Grubbs' Test	Hypothesis test for a single outlier	Confirming a specific point is an outlier	Normal distribution, one outlier at a time	Only tests one point per run
Control Charts (Shewhart)	Mean ± 3σ boundaries on sequential data	Manufacturing, SPC, stable processes	Stationary process	Doesn't handle seasonality or trend

When to use Tier 1: You have a single metric, the distribution is understandable, and you can eyeball what "normal" looks like. Most operational alerting starts here.

Tier 2: Time Series Methods

Use when your data has trend, seasonality, or autocorrelation that invalidates simple statistical bounds.

Method	How It Works	Best For	Assumptions	Limitations
Moving Average ± Band	Rolling mean ± k · rolling std	Slow-moving metrics with noise	Local stationarity	Lags behind sudden shifts
Exponential Smoothing (ETS)	Weighted average favoring recent observations	Trended or seasonal univariate series	Consistent trend/seasonality pattern	Struggles with abrupt regime changes
STL Decomposition + Residual	Decompose into trend + seasonal + residual; flag large residuals	Strongly seasonal data (daily/weekly patterns)	Additive or multiplicative seasonality	Needs ≥ 2 full seasonal cycles
ARIMA / SARIMA	Fit autoregressive model; flag points outside prediction interval	Stationary or differenced series	Linear relationships, stationary residuals	Requires parameter tuning (p, d, q)
Prophet	Additive model with trend, seasonality, holidays	Business metrics with known calendar effects	Additive components	Can overfit with little data; slow on large series
CUSUM / EWMA Charts	Accumulate deviations from target; signal when cumulative sum exceeds threshold	Detecting small, persistent shifts	Stationary baseline	Not designed for sudden spikes (better for drift)

When to use Tier 2: Your metric has daily, weekly, or seasonal patterns. A flat threshold would fire every Monday morning or every holiday.

Tier 3: Machine Learning Methods

Use when the data is multivariate, the patterns are complex, or statistical methods can't capture what "normal" means.

Method	How It Works	Best For	Pros	Cons
Isolation Forest	Randomly partitions feature space; anomalies are isolated in fewer splits	Tabular multivariate data	No distribution assumptions, handles high dimensionality	Requires training data, not inherently temporal
Local Outlier Factor (LOF)	Compares local density of a point to its neighbors	Clusters of varying density	Detects local anomalies that global methods miss	Expensive on large datasets; sensitive to k
One-Class SVM	Learns a boundary around "normal" data	Small datasets with clean normal examples	Works in high dimensions	Sensitive to kernel choice, slow on large data
Autoencoder (Neural)	Learns to reconstruct normal data; high reconstruction error = anomaly	Complex, high-dimensional data (images, logs, embeddings)	Captures non-linear patterns	Needs significant normal data; opaque
DBSCAN	Density-based clustering; points not in any cluster are anomalies	Spatial or clustered data	No need to specify cluster count	Sensitive to epsilon and min_samples

When to use Tier 3: Simple methods produce too many false positives, the data is multivariate, or "normal" is a complex region in feature space — not a simple range.

---

Selection Strategy

Follow this decision tree to choose a method:

1. Is the data univariate (single metric)?
   ├── Yes → Is there seasonality or trend?
   │         ├── No  → Tier 1: Z-Score, IQR, or Control Chart
   │         └── Yes → Tier 2: STL + Residual, SARIMA, or Prophet
   └── No (multivariate) → 
       Is labeled anomaly data available?
       ├── Yes → Supervised classifier (beyond this skill's scope)
       └── No  → Tier 3: Isolation Forest, LOF, or Autoencoder

2. How much data is available?
   ├── < 100 points   → Tier 1 only (not enough for ML)
   ├── 100–10,000     → Tier 1 or Tier 2
   └── > 10,000       → Any tier is viable

3. What's the cost of a false positive vs. false negative?
   ├── FP is expensive (alert fatigue, manual investigation)
       → Favor higher thresholds, conservative methods (IQR, Grubbs')
   └── FN is expensive (missed fraud, missed outage)
       → Favor lower thresholds, ensemble methods

---

Process

Step 1: Characterize the Data

Before selecting a method, understand what you're working with:

## Data Profile

**Source**: <e.g., "Prometheus metric: api_request_latency_p99">
**Type**: <univariate | multivariate> — <time series | tabular | event stream>
**Volume**: <row/point count, time range>
**Frequency**: <per-second | per-minute | daily | irregular>
**Distribution**: <roughly normal | skewed | heavy-tailed | multimodal | unknown>
**Stationarity**: <stationary | trend | seasonal | both | unknown>
**Known patterns**: <e.g., "higher on weekdays, spikes during deploys">
**Labeled anomalies available?**: <yes (n examples) | no>

Step 2: Define What "Anomalous" Means

An anomaly is domain-specific. Define it before applying any method:

## Anomaly Definition

**Point anomaly**: A single observation that deviates significantly
  → Example: "latency > 500ms when baseline is 50ms"

**Contextual anomaly**: Normal in one context, abnormal in another
  → Example: "1000 rps is normal at 2pm, anomalous at 3am"

**Collective anomaly**: A sequence of observations that together are unusual
  → Example: "5 consecutive requests from the same IP to /admin"

**Detection goal**: <Which type(s) are we looking for?>
**Action on detection**: <Alert | Log | Auto-remediate | Queue for investigation>

Step 3: Establish the Baseline

Build a model of "normal" before looking for deviations:

• For statistical methods: Compute mean, median, std dev, percentiles on a clean reference period (no known incidents)
• For time series methods: Decompose into trend + seasonality + residual on historical data
• For ML methods: Train on a dataset of known-good observations

## Baseline

**Reference period**: <date range used for baseline>
**Exclusions**: <removed known incidents, maintenance windows, deploys>
**Statistics**: mean=<X>, median=<X>, std=<X>, p50=<X>, p95=<X>, p99=<X>
**Seasonality**: <daily cycle? weekly? none?>

Step 4: Select and Apply the Method

Choose a method from the tiers above using the selection strategy. Document the choice:

## Method Selection

**Method**: <name>
**Tier**: <1 | 2 | 3>
**Why this method**: <reasoning tied to data profile>
**Parameters**: <e.g., "Z-score threshold = 3.0", "IQR multiplier = 1.5", "Isolation Forest contamination = 0.01">
**Alternatives considered**: <what else was viable and why it wasn't chosen>

Step 5: Evaluate Results

Every detection method needs calibration. Check these before trusting results:

Check	What to Look For
False positive rate	Are flagged points actually anomalous? Manually inspect a sample.
False negative rate	Are known anomalies being caught? Test against historical incidents.
Threshold sensitivity	How much do results change with ±10% threshold adjustment?
Temporal stability	Does the method degrade over time as data distribution shifts?
Edge behavior	Does it flag every Monday? Every deploy? Every restart? (Contextual blindness)

## Evaluation

**Flagged anomalies**: <count>
**Manually verified (sample of N)**: <true positive rate>
**Known incidents caught**: <X of Y>
**False positive pattern**: <e.g., "flags every deploy; needs deploy-window exclusion">
**Recommended threshold adjustment**: <raise / lower / add context filter>

Step 6: Operationalize (If Building a Pipeline)

For ongoing detection (not one-time analysis):

## Operational Design

**Detection frequency**: <real-time | every 5 min | hourly | daily>
**Baseline refresh**: <static | rolling window of N days | retrain weekly>
**Alert routing**: <PagerDuty | Slack channel | log only>
**Suppression rules**: <e.g., "suppress during deploy windows", "deduplicate within 30 min">
**Escalation**: <when does a detected anomaly become an incident?>
**Dashboard**: <link to Grafana / CloudWatch / custom dashboard>

---

Output Format

Every anomaly detection analysis should produce:

1. Data Profile — What the data looks like
2. Anomaly Definition — What counts as anomalous in this domain
3. Baseline — What "normal" looks like, with statistics
4. Method Selection — Which method, why, and with what parameters
5. Results — What was flagged, with evaluation of accuracy
6. Operational Design — How to run this ongoing (if applicable)

---

Combining Methods

Real-world pipelines often layer methods for better results:

Pattern	How It Works	When to Use
Decompose + Threshold	STL/seasonal decomposition → Z-score or IQR on residuals	Seasonal data where raw Z-scores fire on every peak
Ensemble voting	Run 2–3 methods; flag only when ≥ 2 agree	Reducing false positives when no single method is reliable
Tiered detection	Tier 1 for fast screening → Tier 3 on flagged windows	High-volume data where ML on every point is too expensive
Context layering	Statistical method + deploy/event calendar → suppress known causes	Operational alerting where deploys and maintenance cause expected spikes

Rule of thumb: Don't combine methods to increase sensitivity — combine them to increase precision (fewer false positives).

---

Tooling Reference

Library	Language	Best For
scipy.stats	Python	Z-scores, Grubbs', distribution tests
statsmodels	Python	STL decomposition, ARIMA, ETS, control charts
scikit-learn	Python	Isolation Forest, LOF, One-Class SVM, DBSCAN
prophet	Python	Business metrics with calendar seasonality
pyod	Python	Unified API for 30+ anomaly detection algorithms
adtk	Python	Time series anomaly detection toolkit
ruptures	Python	Change point detection
Grafana ML	Dashboard	Built-in anomaly detection on Prometheus/InfluxDB metrics

Use the statistics skill for assumption validation (normality tests, distribution checks) before applying Tier 1 methods.

---

Common Pitfalls

Pitfall	Why It Fails
Training on dirty data	If the baseline includes anomalies, the model learns them as normal
Flat thresholds on seasonal data	A fixed "alert at > 100ms" fires every peak hour and gets ignored
Ignoring concept drift	A model trained 6 months ago may flag today's normal traffic as anomalous
Over-alerting	50 alerts per day = zero alerts acted on. Fewer, high-confidence alerts win.
Skipping manual inspection	Auto-detected anomalies must be spot-checked. Precision matters.
Using ML when Z-scores work	Complexity for its own sake adds maintenance burden and opacity
Confusing anomaly with root cause	Detection tells you something is unusual, not why. Investigation is a separate step.
No baseline exclusions	Including Black Friday in baseline data makes next week look anomalous