Ecg Classification Macro F1 Eval

ecg-classification-macro-f1-eval

Masked Transformer for Electrocardiogram Classification — Zhou et al. (2023) (arXiv:2309.07136, 2023)

What this evaluates

Evaluates a model's ability to classify 12-lead electrocardiogram (ECG) signals into multiple clinical diagnoses. It probes the model's robustness to class imbalance and its capacity to learn from long, redundant time-series sequences using self-supervised pre-training and supervised fine-tuning.

Datasets

• Fujiak — total 220251; splits: train (-1), val (-1), test (-1)
• PCinC — total 79574; splits: train (-1), val (-1), test (-1)
• PTB-XL — total 21836; splits: train (-1), val (-1), test (-1)

Metrics

• macro F1 score (primary) — range: [0, 1]
  • Harmonic mean of sensitivity (recall) and precision, averaged across all classes (macro average). Computed on the testing set.

Input / output format

Input: 12-lead ECG recordings sampled at 500 Hz over 10 seconds, segmented into chunks of size 25 to form a sequence of length T=200.

Output: Multi-label classification predictions (binary indicators for each of the 22-25 clinical diagnosis labels).

Scoring recipe

def compute_macro_f1(y_true, y_pred):
    f1_scores = []
    for c in range(y_true.shape[1]):
        tp = np.sum((y_true[:, c] == 1) & (y_pred[:, c] == 1))
        fp = np.sum((y_true[:, c] == 0) & (y_pred[:, c] == 1))
        fn = np.sum((y_true[:, c] == 1) & (y_pred[:, c] == 0))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    return np.mean(f1_scores)

Common pitfalls

• Using micro or weighted averaging instead of macro averaging, which masks poor performance on rare classes in this highly imbalanced multi-label setting.
• Ignoring the dataset-specific split protocols (e.g., patient-based 8:1:1 split for Fujiak, 10-fold split for PTB-XL), which can cause data leakage if not strictly followed.
• Applying the same hyperparameters for pre-training and fine-tuning; the paper specifies distinct learning rate schedules, momentum values, and regularization (DropPath, layer-wise LR decay) for each phase.

Evidence (verbatim from paper)

Therefore, we use the macro F1 score as the major evaluation metric throughout this paper.

Citation

@misc{zhou2023maskedtransformer,
  title={Masked Transformer for Electrocardiogram Classification},
  author={Zhou et al. (2023)},
  year={2023},
  note={arXiv:2309.07136}
}

• arXiv: 2309.07136