Pheno Ca Eval

pheno-ca-eval

Neural scaling laws for phenotypic drug discovery — Linsley et al. (2023) (arXiv:2309.16773, 2023)

What this evaluates

Evaluates deep neural networks on phenotypic drug discovery tasks using high-content screening images. It probes the model's ability to deconvolve mechanisms of action, molecular targets, and compound identities from cellular phenotypes, as well as zero-shot compound retrieval for CRISPR perturbations.

Datasets

• Pheno-CA — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted classes out of total predictions for MoA, target, and molecule deconvolution tasks.
• CCE — range: [0, ∞]
  • Standard categorical cross-entropy computed on molecule identity predictions.
• rank-order — range: other
  • Rank-order of representational distance between CRISPR perturbation phenotypes and molecule phenotypes to find matching targets.

Input / output format

Input: High-content screening (iHCS) images of cells perturbed by compounds. For compound discovery, also includes images of CRISPR-perturbed targets.

Output: Class labels for MoA (1,282 classes), target (942 classes), or molecule identity (2,919 classes). For compound discovery, a ranked list of compounds based on representational distance.

Scoring recipe

def compute_accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold)

def compute_cce(preds, gold):
    return -sum(g * log(p) for p, g in zip(preds, gold)) / len(gold)

def compute_rank_order(query_rep, cand_reps, gold_idx):
    dists = [cosine_dist(query_rep, c) for c in cand_reps]
    return sorted(range(len(dists)), key=lambda i: dists[i]).index(gold_idx)

Common pitfalls

• IBP-pretrained models are evaluated with frozen weights and a separate 3-layer MLP probe, not fine-tuned end-to-end on the target task.
• Scaling laws are derived using out-of-distribution (OOD) molecules, not the in-distribution Pheno-CA training set.
• Compound discovery is a zero-shot retrieval task; models are not trained on CRISPR perturbation labels.

Evidence (verbatim from paper)

Each DNN in our zoo was given images of cells perturbed by different compounds, and trained to predict the MoA of a given compound out of 1,282 possibilities (Fig.1a). DNNs were either supervised directly for MoA deconvolution or pretrained with IBP (Fig.2a). Next, DNN weights were frozen and three-layer MLP probes were used to transform image representations from both models into MoA predictions (i.e. there was no direct task supervision for IBP models). Our DNN zoo yielded a wide range of performances on this task. At the low end was a 12.09% accurate 12-layer and 128-feature DNN trained with IBP on 100% of out-of-distribution molecules but only 0.01% of the replicates of each compound. At the high-end was a 52.62% accurate 9-layer and 1512-feature DNN trained with IBP on 100% of out-of-distribution molecules and 75% of the replicates of each compound.

Citation

@misc{linsley2023neural,
  title={Neural scaling laws for phenotypic drug discovery},
  author={Linsley et al. (2023)},
  year={2023},
  note={arXiv:2309.16773}
}

• arXiv: 2309.16773