Hescape Eval

hescape-eval

A Large-Scale Benchmark of Cross-Modal Learning for Histology and Gene Expression in Spatial Transcriptomics — Gindra et al. (2025) (arXiv:2508.01490, 2025)

What this evaluates

Evaluates cross-modal alignment between histology images and spatial gene expression profiles, and tests downstream capabilities in gene mutation classification and direct gene expression prediction from whole-slide images.

Datasets

• 5K — total ?; splits: train (-1), test (-1); repo https://github.com/peng-lab/hescape
• Multi — total ?; splits: train (-1), test (-1); repo https://github.com/peng-lab/hescape
• ImmOnc — total ?; splits: train (-1), test (-1); repo https://github.com/peng-lab/hescape
• Colon — total ?; splits: train (-1), test (-1); repo https://github.com/peng-lab/hescape
• Breast — total ?; splits: train (-1), test (-1); repo https://github.com/peng-lab/hescape
• Lung — total ?; splits: train (-1), test (-1); repo https://github.com/peng-lab/hescape

Metrics

• Recall@5 (primary) — range: [0, 1]
  • Fraction of relevant gene expression profiles retrieved within the top 5 ranked predictions. Calculated as |retrieved_top5 ∩ relevant| / |relevant|.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for binary mutation status classification. Computed per biomarker and averaged or reported per class.
• Pearson correlation coefficient (PCC) — range: [-1, 1]
  • Linear correlation between predicted and actual continuous gene expression values across the test set.
• mean squared error (MSE) — range: [0, ∞)
  • Average of squared differences between predicted and actual gene expression values.

Input / output format

Input: Histology image patches or whole-slide images paired with spatial gene expression profiles (retrieval); histology images alone (mutation/expression prediction).

Output: Ranked list of gene profiles (retrieval); binary mutation status labels (classification); continuous gene expression values (regression).

Scoring recipe

def recall_at_5(retrieved, relevant, k=5):
    return len(set(retrieved[:k]) & set(relevant)) / len(relevant)

def f1_score(y_true, y_pred):
    tp = sum((y_true == 1) & (y_pred == 1))
    fp = sum((y_true == 0) & (y_pred == 1))
    fn = sum((y_true == 1) & (y_pred == 0))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def pcc_mse(y_true, y_pred):
    pcc = np.corrcoef(y_true, y_pred)[0, 1]
    mse = np.mean((y_true - y_pred) ** 2)
    return pcc, mse

Common pitfalls

• Batch effects in gene expression data can skew cross-modal alignment and degrade downstream prediction performance.
• Contrastive pretraining may discard morphological features crucial for unimodal tasks like gene expression prediction.
• Performance gains are highly biomarker-specific and may not generalize across all genomic abnormalities.
• Limited spatial transcriptomics samples (e.g., only one Xenium sample per panel for LUAD) restrict pan-cancer generalization claims.

Evidence (verbatim from paper)

Our large-scale pretraining benchmark demonstrates that the gene encoder DRVI emerges as the key determinant of performance improvement across all tissue panels and both image-to-gene (I2G) and gene-to-image (G2I) metrics ([Tab. 2] and Supplementary[Tab. 7]-[Tab. 8]). Notably, DRVI paired with the image encoders Gigapath, H0mini, and UNI consistently achieves the top-performing Recall@5 Scores across datasets.

Citation

@misc{gindra2025hescape,
  title={A Large-Scale Benchmark of Cross-Modal Learning for Histology and Gene Expression in Spatial Transcriptomics},
  author={Gindra et al. (2025)},
  year={2025},
  note={arXiv:2508.01490}
}

• arXiv: 2508.01490