Protein Mutational Effect Eval

protein-mutational-effect-eval

Multi-level Protein Representation Learning for Blind Mutational Effect Prediction — Yang et al. (2023) (arXiv:2306.04899, 2023)

What this evaluates

Evaluates a model's ability to predict the functional or stability impact of amino acid substitutions in proteins without prior experimental data for the specific variant. It probes zero-shot generalization across diverse protein families, taxonomic groups, and mutational depths (single-site vs. deep mutations).

Datasets

• DTm — total 2967; splits: test (-1)
• DDG — total 2967; splits: test (-1)
• ProteinGym — total ?; splits: test (-1)

Metrics

• TPR@threshold (primary) — range: [0, 1]
  • True Positive Rate at specified percentile thresholds (5%, 25%, 50%). Measures the fraction of truly top-performing mutants correctly identified in the model's top-k predictions, where k is determined by the threshold percentage of the total dataset size.
• Spearman's rho — range: [-1, 1]
  • Rank-based correlation coefficient measuring the monotonic relationship between predicted mutational effect scores and experimentally measured values. Computed separately by mutational depth (single, double, all) and taxon (prokaryote, human, eukaryote, virus).

Input / output format

Input: Protein domain represented as a kNN graph where nodes are residues with features extracted from a frozen ESM2-t33 prefix model, and edges encode spatial topology inferred by a 6-layer EGNN.

Output: Continuous predicted score representing the mutational effect (e.g., change in melting temperature ΔTm, change in Gibbs free energy ΔΔG, or relative fitness).

Scoring recipe

def compute_tpr(pred_scores, true_scores, pct):
    k = max(1, int(len(true_scores) * pct / 100))
    top_k_true = set(np.argsort(true_scores)[-k:])
    top_k_pred = set(np.argsort(pred_scores)[-k:])
    return len(top_k_true & top_k_pred) / len(top_k_true)

def compute_spearman_rho(pred_scores, true_scores):
    return scipy.stats.spearmanr(pred_scores, true_scores).correlation

Common pitfalls

• TPR thresholds (5%, 25%, 50%) refer to percentile cutoffs for top-k selection, not standard precision/recall operating points.
• ProteinGym evaluation explicitly excludes the longest protein (A0A140D2T1_ZIKV_Sourisseau_growth_2019) due to AlphaFold2 folding failure, which must be accounted for in dataset construction.
• Node feature choice (ESM2 version) critically impacts performance; using larger models like t36 can degrade results compared to t33 due to over-parameterization or distribution shift.

Evidence (verbatim from paper)

Table 1 evaluates 100 protein assays using TPR at 5%, 25%, and 50%, wherein P13LG consistently outperforms competitors of varying model sizes. To further examine how our model efficiently achieves top performance relative to other large models, Figure 2 visualizes Spearman’s correlation from predictions of pre-trained models at different model scales.

Citation

@misc{tan2023p13lg,
  title={Multi-level Protein Representation Learning for Blind Mutational Effect Prediction},
  author={Yang et al. (2023)},
  year={2023},
  note={arXiv:2306.04899}
}

• arXiv: 2306.04899