Clinical Turing Test Eval

clinical-turing-test-eval

MedalCare-XL: 16,900 healthy and pathological 12 lead ECGs obtained through electrophysiological simulations — Gillette et al. (2022) (arXiv:2211.15997, 2022)

What this evaluates

Evaluates the physiological realism and clinical fidelity of synthetic 12-lead ECGs by measuring how often expert clinicians can correctly distinguish them from real clinical recordings, and how accurately they can diagnose specific pathologies in both synthetic and real signals.

Datasets

• MedalCare-XL — total 16900; splits: test (800)

Metrics

• accuracy (primary) — range: percent
  • Accuracy = (Number of correctly classified signals) / (Total number of signals evaluated). For pathological cases, it also tracks diagnosis accuracy per pathology class.

Input / output format

Input: 12-lead ECG time-series signals (both synthetic and real clinical recordings) presented without labels.

Output: Clinician's binary classification (synthetic vs. real/clinical) and, for pathological cases, a specific pathology diagnosis.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100

Common pitfalls

• Synthetic signals often lack physiological noise, causing clinicians to misclassify them as real.
• Morphological artifacts like fractionation, improper R-wave propagation, or spiking T-waves lead to false synthetic classifications.
• Pathology misdiagnosis is frequent, e.g., LAO, 1AVB, and MI often mistaken for normal sinus rhythm, while LBBB/RBBB mistaken for MI.

Evidence (verbatim from paper)

The six clinicians correctly classified 464 of the 600 cases, which corresponds to an accuracy of 77.33%. On the other side, 136 signals (22.67%) could not be correctly classified, including 62 (10.34%) synthetic and 74 (12.33%) measured ECGs... The two clinicians correctly classified the signals as either measured or clinical in 166 of the 200 cases, which corresponds to an overall accuracy of 83%.

Citation

@misc{gillette2022medalcarexl,
  title={MedalCare-XL: 16,900 healthy and pathological 12 lead ECGs obtained through electrophysiological simulations},
  author={Gillette et al. (2022)},
  year={2022},
  note={arXiv:2211.15997}
}

• arXiv: 2211.15997