Protap Eval

protap-eval

Protap: A Benchmark for Protein Modeling on Realistic Downstream Applications — Yan et al. (2025) (arXiv:2506.02052, 2025)

What this evaluates

Evaluates protein language models and geometric deep learning architectures on five realistic downstream biological tasks, including binding affinity prediction, functional annotation, mutation effects, cleavage site detection, and PROTAC interaction modeling. It probes how pretraining objectives, structural information integration, and domain-specific inductive biases affect generalization on limited biological data.

Datasets

• Protap Benchmark — total ?; splits: train (-1), test (-1); repo https://github.com/Trust-App-AI-Lab/protap

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the probability that a classifier ranks a random positive instance higher than a random negative instance.
• AUPR — range: [0, 1]
  • Area Under the Precision-Recall Curve. Summarizes the trade-off between precision and recall across all classification thresholds.
• Accuracy — range: [0, 1]
  • Percentage of correctly classified instances out of the total number of instances.
• MSE — range: [0, ∞)
  • Mean Squared Error. Average of the squares of the errors between predicted and actual continuous values.
• Pearson correlation — range: [-1, 1]
  • Pearson correlation coefficient. Measures the linear correlation between predicted and actual values.
• Fmax — range: [0, 1]
  • Maximum F1 score across all classification thresholds. Represents the best harmonic mean of precision and recall achievable by threshold tuning.

Input / output format

Input: Protein sequences and/or 3D atomic coordinates. For interaction tasks (PLI, PROTACs), molecular/ligand structures (drugs, warheads, linkers, E3-ligands) are also provided and encoded via GVP.

Output: Task-specific predictions: binary/multi-class labels for cleavage and PROTAC tasks, continuous binding affinity scores for PLI, functional ontology labels for PFA, and mutation effect scores for MTP.

Scoring recipe

def compute_metrics(preds, golds):
    auc = roc_auc_score(golds, preds)
    aupr = average_precision_score(golds, preds)
    acc = accuracy_score(golds, (preds > 0.5).astype(int))
    mse = mean_squared_error(golds, preds)
    pearson = pearsonr(golds, preds)[0]
    fmax = max_f1_score(golds, preds)
    return {'AUC': auc, 'AUPR': aupr, 'Accuracy': acc, 'MSE': mse, 'Pearson': pearson, 'Fmax': fmax}

Common pitfalls

• Using different train/test splits across models instead of the fixed dataset per task specified in the paper.
• Incorrectly unfreezing pretrained encoders during fine-tuning; the protocol requires frozen encoders with only task-specific heads updated.
• Omitting molecular encoders (GVP) for ligand/warhead/linker components in interaction tasks, which breaks consistency with the benchmark setup.

Evidence (verbatim from paper)

Table 4: Performance comparison across model architectures under different training strategies. The first line for each model (e.g., EGNN) denotes a randomly initialized protein encoder trained purely with downstream task supervision. The subsequent lines (e.g., w/MLM) represent pretrained encoders with frozen weights, where only the task-specific head is fine-tuned. C14.005AUC(%)↑ AUPR(%)↑ M10.003AUC(%)↑ AUPR(%)↑ PROTACDBAcc(%)↑ AUPR(%)↑ DavisMSE↓ Pear(%)↑ MFFmax(%)↑ AUPR(%)↑ DMSPear(%)↑ AUPR(%)↑

Citation

@misc{yan2025protap,
  title={Protap: A Benchmark for Protein Modeling on Realistic Downstream Applications},
  author={Yan et al. (2025)},
  year={2025},
  note={arXiv:2506.02052}
}

• arXiv: 2506.02052