Copep Eval

copep-eval

CoPeP: Benchmarking Continual Pretraining for Protein Language Models — Patil et al. (2026) (arXiv:2603.00253, 2026)

What this evaluates

Evaluates the ability of protein language models to adapt to evolving biological databases through continual pretraining. It probes how well models maintain performance on high-quality sequence validation, predict mutation fitness effects, and generalize across diverse protein understanding tasks over time.

Datasets

• UniProt Validation Set — total ?; splits: test (-1)
• ProteinGym — total ?; splits: test (-1)
• PEER — total ?; splits: test (-1)
• DGEB — total ?; splits: test (-1)

Metrics

• Perplexity — range: other
  • Standard language model perplexity computed as the exponential of the average negative log-likelihood of the ground-truth amino acid sequence under the model's predicted distribution.
• Sequence Recovery — range: [0, 1]
  • The proportion of correctly predicted amino acid positions relative to the total number of positions in the sequence.
• Spearman correlation (primary) — range: other
  • Rank-based correlation coefficient measuring the monotonic relationship between predicted fitness effects and experimentally observed fitness values.
• Win rate — range: [0, 1]
  • The proportion of times a given model outperforms all other competing models across every task and training year in the benchmark suite.

Input / output format

Input: Amino acid sequences (strings) representing protein data, often provided as single sequences for language modeling or paired with mutation/context information for downstream fitness and understanding tasks.

Output: Model predictions including next-token probabilities, sequence recovery matches, continuous fitness scores, or discrete task labels, depending on the specific benchmark task.

Scoring recipe

def compute_metrics(predictions, gold, metric_name):
    if metric_name == 'perplexity':
        return exp(-mean(log(predictions)))
    elif metric_name == 'sequence_recovery':
        return mean(predictions == gold)
    elif metric_name == 'spearman':
        return spearmanr(predictions, gold).correlation
    elif metric_name == 'win_rate':
        wins = sum(1 for p, g in zip(predictions, gold) if p > g)
        return wins / len(predictions)
    return None

Common pitfalls

• Temporal Replay optimizes for historical sequence retention, causing it to significantly underperform on ProteinGym which requires predicting heterogeneous mutation effects rather than preserving past data.
• Win rate aggregation across PEER and DGEB masks task-specific performance variations; researchers must consult fine-grained appendix results for detailed analysis.
• Joint training baselines may appear suboptimal not due to continual learning failure, but because they are trained on redundant/pseudogene sequences later removed from UniProt, unlike continual models that filter them out.

Evidence (verbatim from paper)

The perplexity and sequence recovery on the UniProt validation set shown in Figure 5 show that the performance of the continual learning baselines generally improves over time.

Citation

@misc{patil2026copep,
  title={CoPeP: Benchmarking Continual Pretraining for Protein Language Models},
  author={Patil et al. (2026)},
  year={2026},
  note={arXiv:2603.00253}
}

• arXiv: 2603.00253