Psbench Eval

psbench-eval

PSBench: a large-scale benchmark for estimating the accuracy of protein complex structural models — Neupane et al. (2025) (arXiv:2505.22674, 2025)

What this evaluates

Evaluates the ability of Estimation of Model Accuracy (EMA) methods to predict the structural quality of protein complex models. It probes global and interface-level accuracy estimation using correlation, ranking, and classification metrics against reference structural scores.

Datasets

• CASP16_inhouse_TOP5_dataset — total 32; splits: test (32); repo https://github.com/BioinfoMachineLearning/PSBench
• CASP16_community_dataset — total 37; splits: test (37); repo https://github.com/BioinfoMachineLearning/PSBench

Metrics

• Pearson’s correlation (CorrP) (primary) — range: [-1, 1]
  • Measures the linear correlation between predicted quality scores and reference scores (e.g., TM-score or DockQ_wave). Ranges from -1 to 1, where 1 indicates perfect positive linear relationship.
• Spearman’s correlation (CorrS) — range: [-1, 1]
  • Measures the monotonic relationship between predicted and reference scores based on rank ordering. Ranges from -1 to 1.
• Ranking loss (Loss) — range: [0, 1]
  • Calculates the fraction of incorrectly ordered pairs of models based on predicted versus reference scores. Lower values indicate better ranking consistency.
• AUROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, evaluating the ability to classify models as high or low quality based on a threshold applied to reference scores.

Input / output format

Input: Protein complex structural models (typically top-ranked predictions from structure predictors) along with target sequence information.

Output: Predicted global quality score (e.g., TM-score) and/or interface quality score (e.g., DockQ_wave) for each model/target.

Scoring recipe

def evaluate(y_pred, y_true):
    pearson = pearsonr(y_pred, y_true).statistic
    spearman = spearmanr(y_pred, y_true).statistic
    n = len(y_pred)
    discordant = sum(1 for i in range(n) for j in range(i+1, n) if (y_pred[i]-y_pred[j])*(y_true[i]-y_true[j]) < 0)
    loss = discordant / (n*(n-1)/2)
    y_true_bin = (y_true > threshold).astype(int)
    auroc = roc_auc_score(y_true_bin, y_pred)
    return pearson, spearman, loss, auroc

Common pitfalls

• Excluding targets with identical sequences but different conformations (e.g., T1249, T1294) to avoid data leakage.
• Excluding very large targets (e.g., H1217, H1227) when computational constraints prevent model generation.
• Evaluating only on the top 5 models per target rather than the full predictor output.

Evidence (verbatim from paper)

In terms of a global quality score - TM-score, GATE-AFM achieved the highest Spearman’s correlation (0.283), the lowest ranking loss (0.102), the best AUROC (0.658), and second highest Pearson’s correlation (0.372), indicating superior ranking consistency and classification reliability.

Citation

@misc{neupane2025psbench,
  title={PSBench: a large-scale benchmark for estimating the accuracy of protein complex structural models},
  author={Neupane et al. (2025)},
  year={2025},
  note={arXiv:2505.22674}
}

• arXiv: 2505.22674