Fettadbench Eval

fettadbench-eval

FedTADBench: Federated Time-Series Anomaly Detection Benchmark — Liu et al. (2022) (arXiv:2212.09518, 2022)

What this evaluates

This benchmark evaluates federated time-series anomaly detection systems by measuring how well models maintain detection accuracy when trained across decentralized clients compared to centralized baselines. It probes the robustness of anomaly detection architectures under federated learning protocols and varying degrees of non-IID data partitioning.

Datasets

• Time-series anomaly detection datasets (specific names not provided in excerpt) — total ?; splits: train (-1), test (-1); repo https://github.com/fanxingliu2020/FedTADBench

Metrics

• detection accuracy (primary) — range: [0, 1]
  • Standard anomaly detection metric computed on test splits, typically AUC-ROC or F1-score, measuring the model's ability to distinguish normal from anomalous time-series segments.
• training efficiency — range: other
  • Measures the computational and communication cost of federated training, typically tracked via convergence time, communication rounds, or epoch count.

Input / output format

Input: Multivariate or univariate time-series sequences partitioned across multiple federated clients, each containing timestamped sensor readings and corresponding ground-truth anomaly labels.

Output: Per-timestep anomaly scores or binary classification labels indicating normal vs. anomalous behavior.

Scoring recipe

def compute_detection_accuracy(predictions, labels):
    tp = sum(p == 1 and l == 1 for p, l in zip(predictions, labels))
    fp = sum(p == 1 and l == 0 for p, l in zip(predictions, labels))
    fn = sum(p == 0 and l == 1 for p, l in zip(predictions, labels))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• Ignoring the impact of non-IID data partitioning across clients, which significantly skews federated performance compared to centralized baselines.
• Evaluating only final model accuracy without tracking federated training efficiency (communication rounds, convergence time) and stability across heterogeneous clients.
• Applying centralized evaluation protocols directly to federated settings without accounting for client dropout or partial data availability.

Evidence (verbatim from paper)

evaluating five representative anomaly detection models across three dimensions: (1) performance degradation under federated learning vs. centralized training, (2) compatibility of four federated learning methods (FedAvg, FedProx, SCAFFOLD, MOON) with anomaly detection models, including training efficiency, and (3) impact of client data heterogeneity (partitioning) on detection accuracy.

Citation

@misc{liu2022fettadbench,
  title={FedTADBench: Federated Time-Series Anomaly Detection Benchmark},
  author={Liu et al. (2022)},
  year={2022},
  note={arXiv:2212.09518}
}

• arXiv: 2212.09518