Tad Bench Eval

tad-bench-eval

TAD-Bench: A Comprehensive Benchmark for Embedding-Based Text Anomaly Detection — Cao et al. (2025) (arXiv:2501.11960, 2025)

What this evaluates

Evaluates the effectiveness of various text embedding models combined with different anomaly detection algorithms for identifying text anomalies. It probes how well embedding-based anomaly detection generalizes across diverse domains (spam, fake news, hate speech) and distinguishes between patterned versus context-dependent anomalies.

Datasets

• Email-Spam — total ?; splits: (unstated)
• SMS-Spam — total ?; splits: (unstated)
• COVID-Fake — total ?; splits: (unstated)
• LIAR2 — total ?; splits: (unstated)
• Hate-Speech — total ?; splits: (unstated)
• OLID — total ?; splits: (unstated)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve; computed as the integral of the True Positive Rate vs. False Positive Rate across all classification thresholds.

Input / output format

Input: Text instances (emails, SMS messages, news articles, social media posts) processed through a chosen embedding model to produce fixed-dimensional vectors.

Output: Continuous anomaly scores or binary predictions for each embedded text instance, aggregated into per-dataset AUROC scores.

Scoring recipe

import numpy as np
from sklearn.metrics import roc_auc_score

def compute_auroc(y_true, y_scores):
    # y_true: binary labels (0=normal, 1=anomalous)
    # y_scores: continuous anomaly scores from detector
    return roc_auc_score(y_true, y_scores)

Common pitfalls

• Assuming larger or more parameter-heavy embeddings always yield better anomaly detection; lightweight models like MINILM often match larger ones on explicit tasks.
• Overlooking the critical interaction between embedding type and detector algorithm; kNN and INNE are robust across tasks, while density-based methods like LOF or iForest struggle in high-dimensional semantic spaces.
• Treating all anomaly types as equivalent; patterned anomalies (spam) cluster distinctly in embedding space, whereas context-dependent ones (hate speech) are dispersed, requiring different evaluation strategies.

Evidence (verbatim from paper)

Table [2] summarizes the performance of various anomaly detectors combined with LLM-derived embeddings across different datasets... many embedding-detector combinations achieve high AUC scores, with several exceeding 0.8. Figure 5: Boxplot of AUCROC scores for anomaly detectors on different embeddings across 6 datasets.

Citation

@misc{cao2025tadbench,
  title={TAD-Bench: A Comprehensive Benchmark for Embedding-Based Text Anomaly Detection},
  author={Cao et al. (2025)},
  year={2025},
  note={arXiv:2501.11960}
}

• arXiv: 2501.11960