Ecg Preprocessing Eval

ecg-preprocessing-eval

Exploring Best Practices for ECG Pre-Processing in Machine Learning — Salimi et al. (2023) (arXiv:2311.04229, 2023)

What this evaluates

Evaluates how ECG signal pre-processing techniques, particularly down-sampling rates, affect the performance of multi-label time-series classification models for diagnosing heart conditions. It probes the trade-off between signal fidelity, computational cost, and diagnostic accuracy across varying sampling frequencies.

Datasets

• Unspecified multi-label ECG datasets — total ?; splits: train (-1), test (-1)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank; for each sample, takes the reciprocal of the rank of the first correctly predicted label in the model's ranked output list, then averages across all samples.
• F1 — range: [0, 1]
  • F1 score; harmonic mean of precision and recall computed per disease label, then averaged across all labels to yield a single multi-label performance value.
• Spearman correlation — range: [-1, 1]
  • Non-parametric measure of monotonic association between two continuous variables (sampling rate in Hz and F1 performance), ranging from -1 to 1.

Input / output format

Input: ECG time-series signals sampled at varying rates (50–500 Hz), optionally processed with band-pass filtering and min-max normalization.

Output: Multi-label binary predictions indicating the presence or absence of specific heart conditions/diseases, typically returned as a ranked list of labels or a binary vector per sample.

Scoring recipe

def compute_mrr(predictions, gold):
    reciprocal_ranks = []
    for pred, true_set in zip(predictions, gold):
        rank = next((i+1 for i, label in enumerate(pred) if label in true_set), len(pred)+1)
        reciprocal_ranks.append(1.0 / rank)
    return sum(reciprocal_ranks) / len(reciprocal_ranks)

def compute_f1(predictions, gold):
    tp = np.sum((predictions == 1) & (gold == 1), axis=0)
    fp = np.sum((predictions == 1) & (gold == 0), axis=0)
    fn = np.sum((predictions == 0) & (gold == 1), axis=0)
    f1_per_label = 2 * tp / (2 * tp + fp + fn + 1e-8)
    return np.mean(f1_per_label)

Common pitfalls

• Focusing exclusively on per-label trends (e.g., Atrial fibrillation) while ignoring the negligible overall Spearman correlation (0.008) across all labels and models.
• Assuming higher sampling rates universally improve diagnostic performance without accounting for the substantial VRAM and training time overhead at 500 Hz.
• Interpreting Kruskal-Wallis p-values as definitive proof of sampling rate importance without adjusting for multiple comparisons across 20 diseases.

Evidence (verbatim from paper)

The MRR for each model when predicting each label is shown in Figure 1. Here, no consistent pattern can be seen across diseases and scaling rates. ... We also show the Spearman correlation between sampling rate and performance in Table 8. ... correlation measures the mapping between two continuous variables: sampling rates between 50 - 500Hz and F1 performance. Overall, when considering all models and all datasets, we see a negligible correlation of 0.008.

Citation

@misc{salimi2023ecgpreprocessing,
  title={Exploring Best Practices for ECG Pre-Processing in Machine Learning},
  author={Salimi et al. (2023)},
  year={2023},
  note={arXiv:2311.04229}
}

• arXiv: 2311.04229