Etp Ecg Linear Zero Shot Eval

etp-ecg-linear-zero-shot-eval

ETP: Learning Transferable ECG Representations via ECG-Text Pre-training — Che Liu et al. (2023) (arXiv:2309.07145, 2023)

What this evaluates

This protocol evaluates the transferability and robustness of pre-trained ECG representations by measuring classification performance on downstream cardiac disease datasets. It probes both supervised linear probing capabilities and cross-modal zero-shot generalization to specific cardiac conditions without fine-tuning the encoder.

Datasets

• PTB-XL — total 21837; splits: train (-1), val (-1), test (-1)
• CPSC2018 — total 6877; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the probability that a classifier ranks a positive example higher than a negative one across all classification thresholds.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Evaluated per class and averaged.
• ACC — range: [0, 1]
  • Classification accuracy, calculated as the number of correct predictions divided by the total number of predictions.

Input / output format

Input: 12-lead ECG signal (500 Hz sampling rate, up to 60s duration) paired with clinical text report. For zero-shot, the input is the ECG signal and a fixed text prompt template 'this ECG indicates [disease_name]'.

Output: Predicted diagnostic category label (e.g., NORM, MI, STTC, CD, HYP for PTB-XL; 9 specific arrhythmia/condition labels for CPSC2018). In zero-shot, the category whose prompt embedding has the highest cosine similarity to the ECG embedding.

Scoring recipe

def linear_eval(encoder, classifier, loader):
    encoder.eval()
    preds, labels = [], []
    for x, y in loader:
        with torch.no_grad():
            feats = encoder(x)
        preds.append(classifier(feats).argmax(dim=1))
        labels.append(y)
    return compute_metrics(torch.cat(preds), torch.cat(labels))

def zero_shot_eval(ecg_enc, text_enc, loader, diseases):
    ecg_enc.eval(); text_enc.eval()
    correct = 0
    for x, y in loader:
        with torch.no_grad():
            e_emb = ecg_enc(x)
        prompts = [f'this ECG indicates {d}' for d in diseases]
        t_embs = text_enc(prompts)
        sim = cosine_similarity(e_emb, t_embs)
        if diseases[sim.argmax(dim=1)] == y: correct += 1
    return correct / len(loader)

Common pitfalls

• The evaluation strictly uses the official train/val/test splits provided by the dataset creators, not random splits, which significantly affects comparability with other works.
• Zero-shot classification relies on a fixed prompt template and cosine similarity, which may not generalize well to diseases with highly variable clinical descriptions.
• Linear evaluation freezes the pre-trained ECG encoder entirely, meaning performance reflects representation quality rather than full fine-tuning capability.

Evidence (verbatim from paper)

To do this, we keep the pre-trained ECG encoder fixed and only update a linear classifier that is initialized randomly. This evaluation methodology is applied to two large-scale public ECG datasets with disease-level annotation, PTB-XL and CPSC2018, using Area Under the Curve (AUC) score and F1-score as the primary metrics for performance assessment.

Citation

@misc{liu2023etp,
  title={ETP: Learning Transferable ECG Representations via ECG-Text Pre-training},
  author={Che Liu et al. (2023)},
  year={2023},
  note={arXiv:2309.07145}
}

• arXiv: 2309.07145