Ecg Delineation Eval

ecg-delineation-eval

Deep learning based ECG segmentation for delineation of diverse arrhythmias — Joung et al. (2023) (arXiv:2304.06237, 2023)

What this evaluates

This benchmark evaluates a model's ability to accurately segment and delineate the onset and offset boundaries of P, QRS, and T waves in electrocardiogram (ECG) signals. It specifically probes robustness across diverse cardiac arrhythmias and tests the effectiveness of classification-guided post-processing in reducing false positive detections during atrial fibrillation and flutter.

Datasets

• Internal dataset — total ?; splits: test (-1)
• LUDB — total ?; splits: test (-1)
• QTDB — total ?; splits: test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of sensitivity and positive predictive value: F1 = 2 * (Se * PPV) / (Se + PPV). Reported as a percentage in the paper.
• Sensitivity (Se) — range: [0, 1]
  • True positive rate: Se = TP / (TP + FN).
• Positive Predictive Value (PPV) — range: [0, 1]
  • Precision: PPV = TP / (TP + FP).
• Mean error (m) — range: ms
  • Average time deviation (in milliseconds) of correctly detected points from their ground truth annotations.
• Standard deviation of error (σ) — range: ms
  • Standard deviation of the time deviations for correctly detected points.

Input / output format

Input: Single-lead ECG signal (typically lead II or the lead with the lowest delineation error), resampled to 500 Hz.

Output: Predicted onset and offset timestamps (in milliseconds) for P, QRS, and T waves per cardiac beat.

Scoring recipe

def compute_ecg_metrics(preds, gold, tolerance=150):
    TP, FP, FN = 0, 0, 0
    errors = []
    for p in preds:
        matched = [g for g in gold if abs(p - g) <= tolerance]
        if matched:
            TP += 1
            errors.append(abs(p - min(matched)))
        else:
            FP += 1
    for g in gold:
        if not any(abs(p - g) <= tolerance for p in preds):
            FN += 1
    Se = TP / (TP + FN) if (TP + FN) > 0 else 0
    PPV = TP / (TP + FP) if (TP + FP) > 0 else 0
    F1 = 2 * (Se * PPV) / (Se + PPV) if (Se + PPV) > 0 else 0
    m = sum(errors) / len(errors) if errors else 0
    sigma = (sum((e - m)**2 for e in errors) / len(errors))**0.5 if errors else 0
    return Se, PPV, F1, m, sigma

Common pitfalls

• Evaluations must be repeated 20 times and averaged to match the reported results; single-run scores will not align with the paper.
• QTDB annotation format does not allow exact PPV calculation for some waves because absent manual annotations are treated as non-included rather than true negatives.
• Deep learning-based methods consistently exhibit higher standard deviation of error (σ) compared to wavelet-based methods, particularly for T-wave offsets.
• P-wave detection fails frequently during atrial fibrillation and flutter without classification-guided post-processing, leading to high false positive rates.

Evidence (verbatim from paper)

To ensure soundness, we follow the usual standard chosen by The Association for the Advancement of Medical Instrumentation(AAMI) [[42]], which considers an onset or an offset to be correctly detected if an algorithm locates the same type of annotation in a neighborhood of 150ms. Using this threshold value, we examine for each predicted point whether the prediction correctly detects a point in the ground truth annotation. If a ground truth annotation is correctly detected, we count a true positive(TP). ... Based on this, we calculate the following evaluation metrics: ... sensitivity Se=TP/(TP+FN) ... positive predictive value PPV=TP/(TP+FP) ... F1-score F1=2·(Se·PPV)/(Se+PPV)

Citation

@misc{joung2023ecgsegmentation,
  title={Deep learning based ECG segmentation for delineation of diverse arrhythmias},
  author={Joung et al. (2023)},
  year={2023},
  note={arXiv:2304.06237}
}

• arXiv: 2304.06237