Mina Ecg Af Eval

mina-ecg-af-eval

MINA: Multilevel Knowledge-Guided Attention for Modeling Electrocardiography Signals — Hong et al. (2019) (arXiv:1905.11333, 2019)

What this evaluates

Evaluates deep learning models for binary classification of atrial fibrillation (AF) versus control using single-lead ECG recordings. It probes the model's ability to capture beat-level morphology, rhythm-level dynamics, and frequency-domain patterns for clinical AF detection.

Datasets

• PhysioNet Challenge 2017 — total 8528; splits: train (-1), val (-1), test (-1)

Metrics

• ROC-AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• PR-AUC (primary) — range: [0, 1]
  • Area under the Precision-Recall Curve, measuring the trade-off between precision and recall across thresholds. Explicitly recommended by authors for this imbalanced dataset.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall, calculated as 2 * (precision * recall) / (precision + recall) at a 0.5 decision threshold.

Input / output format

Input: 1D ECG voltage time series of exactly 3000 points (9 seconds), extracted via non-overlapping sliding window segmentation from raw recordings sampled at 300Hz.

Output: Binary classification label: AF patient (1) or control (0).

Scoring recipe

def evaluate(y_true, y_pred_prob):
    y_pred = (y_pred_prob >= 0.5).astype(int)
    roc_auc = roc_auc_score(y_true, y_pred_prob)
    pr_auc = average_precision_score(y_true, y_pred_prob)
    f1 = f1_score(y_true, y_pred)
    return {'ROC-AUC': roc_auc, 'PR-AUC': pr_auc, 'F1': f1}

Common pitfalls

• The dataset is highly imbalanced (738 AF vs 7790 controls); ROC-AUC can be misleading, so PR-AUC is the recommended primary metric.
• Models must be evaluated over 5 independent runs with different random seeds, reporting mean ± standard deviation, rather than a single run.
• Raw recordings vary in length (9s to 60s); inputs must be strictly normalized to 3000 points to match the model's expected shape.

Evidence (verbatim from paper)

Performance was measured by the Area under the Receiver Operating Characteristic (ROC-AUC), Area under the Precision-Recall Curve (PR-AUC) and the F1 score. The PR-AUC is considered a better measure for imbalanced data like ours. Table 3 shows MINA outperforms all baselines, and shows 5.51% higher PR-AUC than the second best models.

Citation

@misc{hong2019mina,
  title={MINA: Multilevel Knowledge-Guided Attention for Modeling Electrocardiography Signals},
  author={Hong et al. (2019)},
  year={2019},
  note={arXiv:1905.11333}
}

• arXiv: 1905.11333