Ppb Affinity Eval

ppb-affinity-eval

Beyond Simple Concatenation: Fairly Assessing PLM Architectures for Multi-Chain Protein-Protein Interactions Prediction — Alsamkary et al. (2025) (arXiv:2505.20036, 2025)

What this evaluates

Evaluates protein language model architectures for predicting binding affinity in multi-chain protein-protein complexes. It probes how well different architectural designs capture inter-chain interactions compared to simple sequence or embedding concatenation.

Datasets

• PPB-Affinity — total ?; splits: val (-1), test (-1); repo https://github.com/Proteinea/ppiseq

Metrics

• Spearman ρ (primary) — range: [-1, 1]
  • Spearman rank correlation coefficient between predicted and true binding affinities.
• Pearson r — range: [-1, 1]
  • Pearson linear correlation coefficient between predicted and true binding affinities.
• RMSE (pKd) — range: [0, ∞)
  • Root mean squared error between predicted and true pKd values, calculated as sqrt(mean((y_pred - y_true)^2)).

Input / output format

Input: Paired protein sequences representing ligand and receptor chains for multi-chain complexes.

Output: A single continuous scalar value representing the predicted binding affinity (pKd).

Scoring recipe

import numpy as np
from scipy.stats import spearmanr, pearsonr

def compute_metrics(y_true, y_pred):
    spearman_rho, _ = spearmanr(y_true, y_pred)
    pearson_r, _ = pearsonr(y_true, y_pred)
    rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
    return spearman_rho, pearson_r, rmse

Common pitfalls

• Data leakage can occur if sequence identity between splits exceeds 30%; the protocol enforces a strict ≤30% identity split.
• Simple concatenation (SC) of all chains into one sequence with a single EOS token obscures ligand-receptor boundaries, diluting inter-chain interaction signals.
• Full fine-tuning of certain PLMs (e.g., ESM3-SM-Open) can cause training instability with exploding hidden states, requiring careful learning rate and normalization adjustments.

Evidence (verbatim from paper)

Metrics (mean ± standard deviation, 3 seeds): Spearman $ ho$ , Pearson r, and RMSE $(pK_d)$ . PAD: Pooled attention addition; HP: Hierarchical pooling

Citation

@misc{alsamkary2025beyond,
  title={Beyond Simple Concatenation: Fairly Assessing PLM Architectures for Multi-Chain Protein-Protein Interactions Prediction},
  author={Alsamkary et al. (2025)},
  year={2025},
  note={arXiv:2505.20036}
}

• arXiv: 2505.20036