# Anomaly Detection Meta Eval

> anomaly-detection-meta-eval

- Skill: `qhjqhj00/anomaly-detection-meta-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/anomaly-detection-meta-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/anomaly-detection-meta-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/anomaly-detection-meta-eval

---


# anomaly-detection-meta-eval

> A Meta-Analysis of the Anomaly Detection Problem — Emmott et al. (2015) (arXiv:1503.01158, 2015)

## What this evaluates

Evaluates anomaly detection algorithms on a large corpus of synthetic datasets systematically varied along four dimensions: point difficulty, semantic variation, relative frequency, and feature relevance. It probes algorithm robustness, generalization across diverse anomaly-generating processes, and the impact of experimental design choices on reported performance.

## Datasets

- **Synthetic Anomaly Detection Corpus** — total 1000; splits: test (1000)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve. It measures the probability that a randomly chosen positive instance ranks higher than a randomly chosen negative instance, integrating the true positive rate against the false positive rate across all thresholds.
- `AP` — range: [0, 1]
  - Average Precision. It computes the area under the precision-recall curve, summarizing the trade-off between precision and recall across all classification thresholds, particularly sensitive to class imbalance.

## Input / output format

**Input**: Feature vectors for data points, comprising normal instances drawn from a real-world generating process and anomalous instances drawn from a distinct process, with controlled variations in difficulty, frequency, clustering, and feature relevance.

**Output**: Continuous anomaly scores or binary anomaly labels for each data point.

## Scoring recipe

```python
def compute_auc(y_true, y_scores):
    desc_idx = np.argsort(y_scores)[::-1]
    y_true_sorted = y_true[desc_idx]
    tpr = np.cumsum(y_true_sorted) / np.sum(y_true)
    fpr = np.cumsum(1 - y_true_sorted) / np.sum(1 - y_true)
    return np.trapz(tpr, fpr)

def compute_ap(y_true, y_scores):
    desc_idx = np.argsort(y_scores)[::-1]
    y_true_sorted = y_true[desc_idx]
    precision = np.cumsum(y_true_sorted) / np.arange(1, len(y_true_sorted) + 1)
    return np.sum(precision * y_true_sorted) / np.sum(y_true)
```

## Common pitfalls

- Using too few datasets causes algorithms to overfit to specific problem instances rather than generalizing.
- Treating points merely in the tails of the normal distribution as anomalies ignores the requirement that anomalies come from a distinct generating process.
- Ignoring feature relevance leads to high-dimensional spaces where density estimation fails and normal points fall into distribution tails.

## Evidence (verbatim from paper)

> Using AUC and AP as primary metrics, it reveals significant performance variability across algorithms and demonstrates how experimental design heavily influences results; statistical hypothesis testing shows that many reported improvements are not statistically significant, while a trivial baseline (e.g., uniform scoring) exposes the lack of meaningful performance gains in most cases, highlighting systemic flaws in current anomaly detection evaluation practices.

## Citation

```bibtex
@misc{emmott2015meta,
  title={A Meta-Analysis of the Anomaly Detection Problem},
  author={Emmott et al. (2015)},
  year={2015},
  note={arXiv:1503.01158}
}
```

- arXiv: 1503.01158