Fed Ecg Eval

fed-ecg-eval

FedCVD: The First Real-World Federated Learning Benchmark on Cardiovascular Disease Data — Zhang et al. (2024) (arXiv:2411.07050, 2024)

What this evaluates

Evaluates federated learning algorithms on ECG classification tasks under non-IID and long-tailed label distribution challenges across multiple medical institutions. It probes how well FL methods generalize across heterogeneous clinical data and handle class imbalance without centralizing all data.

Datasets

• Fed-ECG — total ?; splits: train (-1), test (-1); repo https://github.com/SMILELab-FL/FedCVD

Metrics

• Micro F1-Score (Mi-F1) (primary) — range: percent
  • Harmonic mean of global precision and recall calculated across all classes and samples, expressed as a percentage.
• Mean Average Precision (mAP) — range: percent
  • Mean of the average precision scores computed per class, expressed as a percentage.

Input / output format

Input: ECG signal data from a specific medical institution (client).

Output: Predicted class labels for ECG classification.

Scoring recipe

def compute_mi_f1(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == g)
    fp = sum(1 for p, g in zip(preds, gold) if p != g)
    fn = sum(1 for p, g in zip(preds, gold) if p != g)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) * 100

Common pitfalls

• Evaluating only on the global test set misses the practical LOCAL performance per client, which is crucial for real-world deployment.
• Long-tail performance is often overlooked; Top-K drop and F1-STD are needed to capture class imbalance effects.
• Simulated non-IID partitions are easier than the dataset's natural partitioning, leading to overoptimistic FL benchmarks.

Evidence (verbatim from paper)

Table 2: The performance of different FL methods on Fed-ECG is reported using two metrics: Micro F1-Score (Mi-F1) and Mean Average Precision (mAP), both expressed as percentages (%).

Citation

@misc{zhang2024fedcvd,
  title={FedCVD: The First Real-World Federated Learning Benchmark on Cardiovascular Disease Data},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2411.07050}
}

• arXiv: 2411.07050