Ecg Benchmark Eval

ecg-benchmark-eval

A Comprehensive Benchmark for Electrocardiogram Time-Series — Tang et al. (2025) (arXiv:2507.14206, 2025)

What this evaluates

Evaluates the capability of models to analyze electrocardiogram (ECG) time-series data across four medical tasks: classification, detection, forecasting, and generation. It probes semantic fidelity and diagnostic accuracy in quasi-periodic physiological signals, emphasizing robustness to temporal shifts and class imbalance.

Datasets

• CPSC2018 — total ?; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
• CPSC2019 — total 2000; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
• CPSC2020 — total ?; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
• CPSC2021 — total ?; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
• MITDB — total 48; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
• PTBXL — total 21799; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
• FEPL — total ?; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
• DALIA — total ?; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark
• SST — total ?; splits: train (-1), test (-1); repo https://github.com/ZhijiangTang/ECG-Benchmark

Metrics

• accuracy — range: [0, 1]
  • Proportion of correctly classified ECG recordings out of the total test set. Computed as correct predictions divided by total samples.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for detection tasks. Predictions are matched to ground truth if within a ±70ms temporal window.
• FFD (primary) — range: [0, inf)
  • Feature-based Fréchet Distance. Computes the Fréchet distance between the latent feature distributions of real and predicted/generated ECG sequences, extracted via a transformer encoder mapping. Lower values indicate higher semantic fidelity.

Input / output format

Input: Single-channel ECG time-series recordings resampled to 100 Hz with a fixed length of 500 points. Multi-channel recordings are split into single-channel. Missing data (>25%) is discarded; otherwise, linear interpolation is used.

Output: Classification: discrete disease/arrhythmia class labels. Detection: predicted waveform probability scores with temporal positions. Forecasting: synthetic ECG time-series sequences of 100 points. Generation: full-length synthetic ECG sequences.

Scoring recipe

def compute_metrics(predictions, golds, task):
    if task == 'classification':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task == 'detection':
        tp = sum(1 for p, g in zip(predictions, golds) if abs(p - g) <= 0.070)
        fp = len(predictions) - tp
        fn = len(golds) - tp
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    elif task in ('forecasting', 'generation'):
        real_feats = [transformer_encoder(x) for x in golds]
        pred_feats = [transformer_encoder(x) for x in predictions]
        return frechet_distance(real_feats, pred_feats)

Common pitfalls

• Using Mean Squared Error (MSE) instead of FFD for generation/forecasting, as MSE is highly sensitive to minor temporal shifts and fails to capture clinical semantics in quasi-periodic ECG signals.
• Ignoring the ±70ms matching window for detection tasks, which leads to artificially low recall and F1 scores despite clinically acceptable predictions.
• Failing to address severe class imbalance in detection tasks without adopting F1-score or proper downsampling/oversampling strategies.

Evidence (verbatim from paper)

Our proposed method, PSSM, achieves state-of-the-art performance across all tasks, with an average performance of 0.947 in classification accuracy, 0.820 in detection F1 score, 0.211 in forecasting FFD, and 0.133 in generation FFD.

Citation

@misc{tang2025ecgbenchmark,
  title={A Comprehensive Benchmark for Electrocardiogram Time-Series},
  author={Tang et al. (2025)},
  year={2025},
  note={arXiv:2507.14206}
}

• arXiv: 2507.14206