Cloudano Bench Eval

cloudano-bench-eval

Towards Generalizable Context-aware Anomaly Detection: A Large-scale Benchmark in Cloud Environments — Zou et al. (2025) (arXiv:2508.01844, 2025)

What this evaluates

Evaluates the ability of systems to detect context-aware anomalies in cloud environments by jointly analyzing system logs and performance metrics, and to classify the specific anomaly scenario. It also tests generalization to point-level anomalies using log-only or metric-only data.

Datasets

• CloudAnoBench — total 1252; splits: test (-1)
• BGL — total ?; splits: test (-1)
• Thunderbird — total ?; splits: test (-1)
• HDFS_v1 — total ?; splits: test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Standard for imbalanced anomaly detection.
• Precision — range: [0, 1]
  • Ratio of true positive predictions to all positive predictions: TP / (TP + FP).
• Recall — range: [0, 1]
  • Ratio of true positive predictions to all actual positives: TP / (TP + FN).
• FPR — range: [0, 1]
  • False Positive Rate: ratio of false alarms to all actual negatives: FP / (FP + TN).
• Accuracy — range: [0, 1]
  • Proportion of correctly identified anomaly scenarios: Acc = |{i | ŷ_i = y_i}| / N, where N is total cases.

Input / output format

Input: Multimodal cloud telemetry per instance: system logs (text) and performance metrics (time-series). For baseline comparisons, inputs are restricted to either metrics-only (ML baselines) or logs-only (log-specific baselines).

Output: Binary anomaly decision (anomalous/normal) and, for scenario identification, a categorical label specifying the anomaly scenario type.

Scoring recipe

def compute_metrics(preds, golds):
    tp = sum(p == 1 and g == 1 for p, g in zip(preds, golds))
    fp = sum(p == 1 and g == 0 for p, g in zip(preds, golds))
    fn = sum(p == 0 and g == 1 for p, g in zip(preds, golds))
    tn = sum(p == 0 and g == 0 for p, g in zip(preds, golds))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    fpr = fp / (fp + tn) if (fp + tn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    return {'Precision': prec, 'Recall': rec, 'FPR': fpr, 'F1-score': f1, 'Accuracy': acc}

Common pitfalls

• ML baselines are restricted to metric-only inputs and cannot process log text, making direct comparison with multimodal methods unfair without input alignment.
• Log-only baselines are evaluated on point-anomaly datasets (BGL, Thunderbird, HDFS_v1) but cannot handle the context-aware, multimodal scenarios in CloudAnoBench.
• FPR is explicitly reported alongside F1-score, which is critical for highly imbalanced anomaly detection datasets where accuracy is misleading.

Evidence (verbatim from paper)

For anomaly detection, the task is modeled as a binary classification problem. We report Precision, Recall, FPR, and F1-score, which together capture the reliability of positive alerts, the ability to identify true anomalies, the tendency to produce false alarms, and the overall balance between precision and recall. For scenario identification (SI), we formulate the task as a multi-class classification problem and evaluate performance using accuracy (Acc).

Citation

@misc{zou2025cloudanobench,
  title={Towards Generalizable Context-aware Anomaly Detection: A Large-scale Benchmark in Cloud Environments},
  author={Zou et al. (2025)},
  year={2025},
  note={arXiv:2508.01844}
}

• arXiv: 2508.01844