Mmist Ccrcc Eval

mmist-ccrcc-eval

MMIST-ccRCC: A Real World Medical Dataset for the Development of Multi-Modal Systems — Mota et al. (2024) (arXiv:2405.01658, 2024)

What this evaluates

Evaluates multi-modal fusion and missing data imputation strategies for predicting 12-month survival in clear cell renal cell carcinoma (ccRCC) patients. It probes a model's ability to integrate heterogeneous clinical, genomic, and imaging data while handling severe modality missingness and class imbalance.

Datasets

• MMIST-ccRCC — total 618; splits: train (-1), test (-1)

Metrics

• BAcc (primary) — range: [0, 1]
  • Balanced Accuracy, defined as the average recall for both classes: BAcc = (Recall_positive + Recall_negative) / 2. Recall is calculated as TP / (TP + FN).

Input / output format

Input: Per patient: multi-modal data including CT/MRI radiology scans, Whole Slide Images (WSI) for histopathology, and clinical/genomic features (ClinGen). Missing modalities are indicated by absence or placeholder vectors in the fusion pipeline.

Output: Binary classification label indicating whether the patient survived or died within 12 months.

Scoring recipe

def compute_bacc(y_true, y_pred):
    tp = np.sum((y_true == 1) & (y_pred == 1))
    fn = np.sum((y_true == 1) & (y_pred == 0))
    tn = np.sum((y_true == 0) & (y_pred == 0))
    fp = np.sum((y_true == 0) & (y_pred == 1))
    recall_pos = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    recall_neg = tn / (tn + fp) if (tn + fp) > 0 else 0.0
    return (recall_pos + recall_neg) / 2.0

Common pitfalls

• The dataset has extreme class imbalance (~88% survival), so standard accuracy will be misleadingly high and does not reflect minority class performance.
• Missing data is highly modality-specific (e.g., MRI only 8%, Genomics 74% missing), requiring explicit imputation or fusion strategies rather than simple deletion.
• The paper uses latent representations for fusion rather than raw images, so evaluators must replicate the feature extraction pipeline to match reported scores.

Evidence (verbatim from paper)

Given this extreme imbalance, we opted to evaluate the results in terms of BAcc, i.e., the average Recall for both classes.

Citation

@misc{mota2024mmistccrcc,
  title={MMIST-ccRCC: A Real World Medical Dataset for the Development of Multi-Modal Systems},
  author={Mota et al. (2024)},
  year={2024},
  note={arXiv:2405.01658}
}

• arXiv: 2405.01658