Flowbench Eval

flowbench-eval

Flow-Bench: A Dataset for Computational Workflow Anomaly Detection — Papadimitriou et al. (2023) (arXiv:2306.09930, 2023)

What this evaluates

Evaluates computational workflow anomaly detection by benchmarking models on detecting injected CPU and HDD performance anomalies in distributed workflow execution logs. It tests the ability of tabular, graph, and text-based methods to identify anomalous nodes within Directed Acyclic Graph (DAG) workflow executions.

Datasets

• Flow-Bench — total 6352; splits: test (-1); repo https://github.com/PeSeiDon-Workflows/flowbench

Metrics

• ROC-AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• AP — range: [0, 1]
  • Average Precision, summarizing the precision-recall curve across all classification thresholds.
• Accuracy — range: [0, 1]
  • Ratio of correctly classified instances (normal and anomalous) to the total number of instances.
• Recall@k — range: [0, 1]
  • Proportion of actual anomalous nodes retrieved among the top-k predicted anomalous nodes.

Input / output format

Input: Tabular CSV logs per DAG execution containing job-level execution metrics, infrastructure statistics, and dependency information. The dataset also provides graph (DAG structure) and text (template-generated sentences) representations.

Output: Anomaly label per node: normal, cpu_2, cpu_3, cpu_4, hdd_5, or hdd_10. For binary detection, a flag indicating anomalous vs normal.

Scoring recipe

def compute_rocauc(y_true, y_scores):
    # y_true: binary node labels (0=normal, 1=anomalous)
    # y_scores: predicted anomaly scores
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return auc(fpr, tpr)

def compute_ap(y_true, y_scores):
    precision, recall, _ = precision_recall_curve(y_true, y_scores)
    return auc(recall, precision)

Common pitfalls

• Anomalies are injected at the per-node level, not the whole-DAG level, so evaluation must score or aggregate per node rather than per workflow execution.
• The dataset is highly imbalanced with a vast majority of normal nodes, making accuracy misleading without precision/recall metrics.
• Graph, tabular, and text representations require distinct preprocessing pipelines; mixing them without alignment causes evaluation errors.

Evidence (verbatim from paper)

Table 2: Unsupervised Model Examples

Citation

@misc{papadimitriou2023flowbench,
  title={Flow-Bench: A Dataset for Computational Workflow Anomaly Detection},
  author={Papadimitriou et al. (2023)},
  year={2023},
  note={arXiv:2306.09930}
}

• arXiv: 2306.09930