Flip Benchmark Eval

flip-benchmark-eval

Exploring Large Protein Language Models in Constrained Evaluation Scenarios within the FLIP Benchmark — Mollon et al. (2025) (arXiv:2501.18223, 2025)

What this evaluates

Evaluates the ability of large protein language models to predict protein fitness under constrained, low-data scenarios. It probes mutation-level generalization, overfitting risks, and the impact of model depth and structural information on predictive accuracy across diverse protein families.

Datasets

• FLIP benchmark — total ?; splits: train (-1), test (-1)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error: the average of the squared differences between predicted and actual fitness values. Lower values indicate better predictive accuracy.
• Spearman’s rank correlation coefficient ($ ho$) — range: [-1, 1]
  • Spearman's rho: a rank-based non-parametric measure of the monotonic relationship between predicted and actual fitness values. Values range from -1 to 1, with 1 indicating perfect agreement.

Input / output format

Input: Protein amino acid sequences (and optionally predicted or known 3D structures for structure-aware models like SaProt), paired with ground-truth fitness measurements during training.

Output: A continuous numerical prediction representing the estimated protein fitness value for each input sequence/mutant.

Scoring recipe

import numpy as np
from scipy.stats import spearmanr

def score(predictions, golds):
    preds = np.array(predictions)
    golds = np.array(golds)
    mse = np.mean((preds - golds) ** 2)
    rho, _ = spearmanr(preds, golds)
    return {'MSE': mse, 'Spearman_rho': rho}
# Final reported value is the median across 10 independent runs per split

Common pitfalls

• Deeper models (e.g., 48 layers) exhibit high training correlation but severe test overfitting on small splits, requiring careful early stopping.
• Long sequences are completely masked due to computational limits (e.g., >2700 AA), potentially biasing performance estimates for large proteins.
• Metrics are reported as medians of 10 runs rather than means, which can obscure variance and complicate direct statistical comparisons with other works.

Evidence (verbatim from paper)

In this section, we present the results of each model’s performance on the protein fitness prediction tasks, evaluated using mean squared error (MSE) and Spearman’s rank correlation coefficient. The results shown represent the median values of ten experiments run per split, as the median is more robust to outliers compared to the average.

Citation

@misc{mollon2025exploring,
  title={Exploring Large Protein Language Models in Constrained Evaluation Scenarios within the FLIP Benchmark},
  author={Mollon et al. (2025)},
  year={2025},
  note={arXiv:2501.18223}
}

• arXiv: 2501.18223