Cardiac Cmr Eval

cardiac-cmr-eval

BAAI Cardiac Agent: An intelligent multimodal agent for automated reasoning and diagnosis of cardiovascular diseases from cardiac magnetic resonance imaging — Qu et al. (2026) (arXiv:2604.04078, 2026)

What this evaluates

Evaluates automated segmentation and diagnostic classification capabilities on cardiac magnetic resonance imaging (CMR) sequences. Probes the model's ability to accurately delineate cardiac structures across multiple anatomical views and classify cardiovascular diseases with clinical-grade metrics.

Datasets

• BAAI Cardiac CMR Cohort — total 2413; splits: train (-1), val (-1), test (-1), external_val (-1)

Metrics

• DSC (primary) — range: [0, 1]
  • Dice Similarity Coefficient measures voxel-wise overlap between predicted and ground-truth masks: 2|A∩B|/(|A|+|B|). Ranges from 0 (no overlap) to 1 (perfect match).
• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to discriminate between classes across all classification thresholds. Higher values indicate better diagnostic performance.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used for both segmentation overlap and diagnostic classification balance.

Input / output format

Input: Multi-frame cardiac MRI sequences (SAX cine, 2CH cine, 4CH cine, SAX LGE) provided as 3D volumes or 2D slice stacks.

Output: For segmentation: binary/semantic masks per cardiac structure. For diagnosis: categorical disease labels (e.g., NH, IHD, NICM, or specific subtypes like HCM, DCM).

Scoring recipe

def compute_metrics(preds, gold):
    # Segmentation
    dsc = 2 * np.sum(preds & gold) / (np.sum(preds) + np.sum(gold))
    # Diagnosis
    tp = np.sum((preds == 1) & (gold == 1))
    fp = np.sum((preds == 1) & (gold == 0))
    fn = np.sum((preds == 0) & (gold == 1))
    precision = tp / (tp + fp + 1e-8)
    recall = tp / (tp + fn + 1e-8)
    f1 = 2 * precision * recall / (precision + recall + 1e-8)
    auc = roc_auc_score(gold, preds_prob)
    return {'DSC': dsc, 'F1': f1, 'AUC': auc}

Common pitfalls

• Segmentation test set is very small (n=30), making per-sequence metric averages unstable and prone to high variance.
• Metrics are reported per sequence type (SAX, 2CH, 4CH, LGE) rather than globally averaged, complicating cross-model comparison.
• External validation uses an independent hospital cohort, so domain shift effects are not controlled for in the internal stratified split.

Evidence (verbatim from paper)

To ensure reliable model training and unbiased performance evaluation for the CDS and NICMS tasks, the internal dataset was partitioned into training, internal validation set, and internal test sets in a 7:1:2 ratio using a stratified split strategy that strictly follows the number distribution of CVDs categories. For normal heart (NH) screening, the model achieved an area under the curve (AUC) of 0.980 (95% confidence interval [CI]: 0.956–0.995) and an F1-score of 0.856 (95% CI: 0.782–0.914).

Citation

@misc{qu2026baai,
  title={BAAI Cardiac Agent: An intelligent multimodal agent for automated reasoning and diagnosis of cardiovascular diseases from cardiac magnetic resonance imaging},
  author={Qu et al. (2026)},
  year={2026},
  note={arXiv:2604.04078}
}

• arXiv: 2604.04078