Rxrx3 Core Eval

rxrx3-core-eval

RxRx3-core: Benchmarking drug-target interactions in High-Content Microscopy — Kraus et al. (2025) (arXiv:2503.20158, 2025)

What this evaluates

Evaluates whether representation learning models capture biologically meaningful signals in high-content microscopy images. It probes the model's ability to distinguish drug-induced perturbations from controls, predict zero-shot drug-target interactions, and recover known gene-gene relationships from phenotypic embeddings.

Datasets

• RxRx3-core — total 222601; splits: benchmark (-1); repo https://github.com/curisdictionpharma/EFAAR_benchmarking

Metrics

• average precision (primary) — range: [0, 1]
  • Computes the area under the precision-recall curve for ranking known drug targets against a randomly sampled set of negative genes. Results are aggregated as the median across all compounds and 100 random negative sampling seeds.
• energy distance — range: other
  • Quantifies distributional separation between perturbation embeddings and control embeddings: E = 2/(n1*n2) * sum||x-y|| - 1/n1^2 * sum||x-x|| - 1/n2^2 * sum||y-y||. Equals zero for identical distributions.
• recall — range: [0, 1]
  • Measures the proportion of true gene-gene interactions (from CORUM, hu.MAP, Reactome, StringDB) that fall within the top 5% or bottom 5% of pairwise cosine similarities between gene embeddings.

Input / output format

Input: Batch-aligned phenomics embeddings for 1,674 compounds and 736 CRISPR knockouts. Embeddings are derived by mean-aggregating four tiled 256x256x6 crops from 512x512x6 microscopy images, followed by PCA-CS centering and scaling relative to control samples.

Output: For drug-target benchmark: a ranked list of gene targets per compound based on cosine similarity scores. For perturbation benchmark: a scalar energy distance value per perturbation. For gene-gene benchmark: a recall score over extreme cosine similarity pairs.

Scoring recipe

def compute_drug_target_benchmark(compound_embs, gene_embs, true_targets, n_seeds=100):
    cos_sim = cosine_similarity(compound_embs, gene_embs)
    ap_scores = []
    for _ in range(n_seeds):
        neg_genes = sample_random_genes(gene_embs, exclude=true_targets)
        ranked_targets = rank_by_cosine(cos_sim, neg_genes)
        ap = average_precision(true_targets, ranked_targets)
        ap_scores.append(ap)
    return median(ap_scores)

Common pitfalls

• Failing to apply PCA-CS batch alignment before computing metrics will introduce severe batch effects and invalidate results.
• Not using 100 random negative sampling seeds for the drug-target benchmark leads to high variance and unreliable z-scores against the random baseline.
• Including self-links (gene vs itself) in the gene-gene cosine similarity matrix biases recall scores to 1.0 and must be excluded.

Evidence (verbatim from paper)

To quantify confidence, we use the absolute value of the cosine similarity as a proxy, analogous to the probability score in a classifier. For each compound, we calculate the Area Under the ROC Curve (AUC) and average precision as performance metrics. The final results are summarized by reporting the median AUC and average precision across all compounds, providing a comparison against a random baseline.

Citation

@misc{kraus2025rxrx3core,
  title={RxRx3-core: Benchmarking drug-target interactions in High-Content Microscopy},
  author={Kraus et al. (2025)},
  year={2025},
  note={arXiv:2503.20158}
}

• arXiv: 2503.20158