# Copep Eval

> Evaluates the ability of protein language models to adapt to evolving biological databases through continual pretraining. It probes how well models maintain performance on high-quality sequence validation, predict mutation fitness effects, and generalize across diverse protein understanding tasks over time. Use when the user wants to benchmark on UniProt Validation Set, ProteinGym, PEER, DGEB, or asks about evaluating this task. Reports Spearman correlation.

- Skill: `qhjqhj00/copep-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/copep-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/copep-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/copep-eval

---


# copep-eval

> CoPeP: Benchmarking Continual Pretraining for Protein Language Models — Patil et al. (2026) (arXiv:2603.00253, 2026)

## What this evaluates

Evaluates the ability of protein language models to adapt to evolving biological databases through continual pretraining. It probes how well models maintain performance on high-quality sequence validation, predict mutation fitness effects, and generalize across diverse protein understanding tasks over time.

## Datasets

- **UniProt Validation Set** — total ?; splits: test (-1)
- **ProteinGym** — total ?; splits: test (-1)
- **PEER** — total ?; splits: test (-1)
- **DGEB** — total ?; splits: test (-1)

## Metrics

- `Perplexity` — range: other
  - Standard language model perplexity computed as the exponential of the average negative log-likelihood of the ground-truth amino acid sequence under the model's predicted distribution.
- `Sequence Recovery` — range: [0, 1]
  - The proportion of correctly predicted amino acid positions relative to the total number of positions in the sequence.
- `Spearman correlation` **(primary)** — range: other
  - Rank-based correlation coefficient measuring the monotonic relationship between predicted fitness effects and experimentally observed fitness values.
- `Win rate` — range: [0, 1]
  - The proportion of times a given model outperforms all other competing models across every task and training year in the benchmark suite.

## Input / output format

**Input**: Amino acid sequences (strings) representing protein data, often provided as single sequences for language modeling or paired with mutation/context information for downstream fitness and understanding tasks.

**Output**: Model predictions including next-token probabilities, sequence recovery matches, continuous fitness scores, or discrete task labels, depending on the specific benchmark task.

## Scoring recipe

```python
def compute_metrics(predictions, gold, metric_name):
    if metric_name == 'perplexity':
        return exp(-mean(log(predictions)))
    elif metric_name == 'sequence_recovery':
        return mean(predictions == gold)
    elif metric_name == 'spearman':
        return spearmanr(predictions, gold).correlation
    elif metric_name == 'win_rate':
        wins = sum(1 for p, g in zip(predictions, gold) if p > g)
        return wins / len(predictions)
    return None
```

## Common pitfalls

- Temporal Replay optimizes for historical sequence retention, causing it to significantly underperform on ProteinGym which requires predicting heterogeneous mutation effects rather than preserving past data.
- Win rate aggregation across PEER and DGEB masks task-specific performance variations; researchers must consult fine-grained appendix results for detailed analysis.
- Joint training baselines may appear suboptimal not due to continual learning failure, but because they are trained on redundant/pseudogene sequences later removed from UniProt, unlike continual models that filter them out.

## Evidence (verbatim from paper)

> The perplexity and sequence recovery on the UniProt validation set shown in Figure 5 show that the performance of the continual learning baselines generally improves over time.

## Citation

```bibtex
@misc{patil2026copep,
  title={CoPeP: Benchmarking Continual Pretraining for Protein Language Models},
  author={Patil et al. (2026)},
  year={2026},
  note={arXiv:2603.00253}
}
```

- arXiv: 2603.00253