Eva Immunology Benchmark Eval

eva-immunology-benchmark-eval

EVA: Towards a universal model of the immune system — Bandasack et al. (2026) (arXiv:2602.10168, 2026)

What this evaluates

Evaluates a multimodal foundation model's ability to predict drug efficacy, classify disease endotypes, and align cross-species transcriptomic and histological data for immunology and inflammation research.

Datasets

• I&I benchmark — total ?; splits: test (-1)
• IBDome dataset — total ?; splits: test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve; measures the probability that a randomly chosen positive instance ranks higher than a randomly chosen negative instance.
• accuracy — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances.

Input / output format

Input: Patient-level transcriptomic data (bulk RNA-seq, microarray, pseudobulk scRNA-seq) or histology tile embeddings; for zero-shot tasks, inputs are perturbed patient transcriptomes simulating target up/downregulation.

Output: Continuous efficacy score between 0 and 1 for zero-shot tasks; discrete class labels (e.g., binary inflammation status, Montreal disease course categories, histopathological diagnosis) for classification tasks.

Scoring recipe

def score_zero_shot(patient_exprs, target, healthy_ref):
    scores = []
    for expr in patient_exprs:
        perturbed = simulate_perturbation(expr, target)
        shift = geometric_distance(perturbed, healthy_ref)
        scores.append(normalize_0_1(shift))
    median_score = median(scores)
    return median_score

# Rank drug-disease pairs by median_score
# Compute AUROC against ground_truth_efficacy
auc = compute_auroc(ranked_predictions, ground_truth_efficacy)

Common pitfalls

• BulkRNABert decoder is not publicly available, preventing direct zero-shot efficacy evaluation.
• Drug-disease efficacy labels are incomplete due to limited clinical trial evidence, requiring careful handling of missing data.
• Performance is highly sensitive to the decision threshold (e.g., 0.5) used to convert continuous scores to binary predictions.

Evidence (verbatim from paper)

For each patient in a disease cohort, we simulated the perturbation of relevant drug targets and measured geometrically how the resulting transcriptomic state shifted relative to healthy reference tissue. Each patient received a score between 0 and 1, where higher values indicate greater alignment toward the healthy phenotype, suggesting potential therapeutic benefit. We then computed the median score across all patients for each drug-disease combination. Drug-disease pairs were ranked by their median alignment scores, and we computed AUROC to assess discrimination between efficacious and non-efficacious treatments.

Citation

@misc{bandasack2026eva,
  title={EVA: Towards a universal model of the immune system},
  author={Bandasack et al. (2026)},
  year={2026},
  note={arXiv:2602.10168}
}

• arXiv: 2602.10168