Hest Benchmark Eval

hest-benchmark-eval

HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis — Jaume et al. (2024) (arXiv:2406.16192, 2024)

What this evaluates

Evaluates the ability of histopathology foundation models to predict gene expression levels from H&E-stained whole-slide image patches. It probes the alignment between morphological features and transcriptomic profiles across diverse cancer types and organs.

Datasets

• HEST-Benchmark — total ?; splits: train (-1), test (-1); repo https://github.com/mahmoodlab/hest

Metrics

• Pearson correlation (primary) — range: [-1, 1]
  • Computes the Pearson correlation coefficient between predicted and measured log1p-normalized gene expression values for the top 50 highly variable genes. The final reported score is the mean correlation across all genes and cross-validation folds.

Input / output format

Input: 112×112 μm H&E-stained image patches (224×224 pixels at 20× magnification) from whole-slide images.

Output: Predicted log1p-normalized expression values for the top 50 highly variable genes per patch.

Scoring recipe

def score(predictions, gold):
    # predictions, gold: arrays of shape (n_samples, 50)
    # Compute Pearson correlation for each gene
    corrs = [pearsonr(gold[:, i], predictions[:, i])[0] for i in range(50)]
    return np.mean(corrs)

Common pitfalls

• Avoid patient-level data leakage by using patient-stratified splits instead of random patch-level splits.
• Directly applying Ridge regression to models with varying embedding dimensions unfairly penalizes larger embeddings; PCA reduction to 256 factors is recommended for fair comparison.
• Performance varies significantly across cancer types, with some tasks (e.g., READ, ccRCC) showing inherently low correlation due to biological or technical noise.

Evidence (verbatim from paper)

All tasks are evaluated using the Pearson correlation between the predicted and measured gene expression. We report mean and standard deviation across all folds (or patients). To avoid train/test patient-level data leakage, we use patient-stratified splits, resulting in a k-fold cross-validation, where k is the number of patients.

Citation

@misc{jaume2024hest1k,
  title={HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis},
  author={Jaume et al. (2024)},
  year={2024},
  note={arXiv:2406.16192}
}

• arXiv: 2406.16192