Protein Function Benchmark Eval

protein-function-benchmark-eval

Open-Source Protein Language Models for Function Prediction and Protein Design — Pandi et al. (2024) (arXiv:2412.13519, 2024)

What this evaluates

Evaluates protein language models on classification and regression tasks to assess their capability in predicting protein properties, sub-cellular localization, epitope regions, and mutational effects.

Datasets

• Sub-cellular Localization — total ?; splits: test (-1)
• Membrane Solubility — total ?; splits: test (-1)
• Epitope Region Prediction — total ?; splits: test (-1)
• GB1 Mutational Landscape — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted labels out of total predictions. Calculated as the number of correct predictions divided by the total number of instances.
• Spearman's rank correlation coefficient (primary) — range: [-1, 1]
  • Rank-based measure of statistical dependence between predicted and actual continuous values. Computed as the Pearson correlation coefficient between the rank-ordered predictions and gold labels.

Input / output format

Input: Raw protein amino acid sequences.

Output: For classification tasks: discrete predicted labels (e.g., cellular compartment, soluble/membrane, epitope region). For regression tasks: continuous activity scores.

Scoring recipe

if task_type == 'classification':
    metric = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
elif task_type == 'regression':
    metric = spearmanr(predictions, golds).correlation

Common pitfalls

• Datasets are loaded via DeepChem's built-in utilities rather than explicit downloads, making exact train/val/test splits and preprocessing steps difficult to reproduce without matching library versions.
• The paper notes pretraining was limited to 1M sequences instead of the original 216M, which may cause performance gaps compared to the official ProtBERT baseline.
• Evaluation uses a one-layer MLP on top of embeddings, but exact fine-tuning hyperparameters (learning rate, batch size, epochs) are omitted, hindering exact replication.

Evidence (verbatim from paper)

For each classification task, we recorded the accuracy, while for the regression tasks, we reported Spearman's rank correlation coefficient $(\rho)$ .

Citation

@misc{pandi2024openprotein,
  title={Open-Source Protein Language Models for Function Prediction and Protein Design},
  author={Pandi et al. (2024)},
  year={2024},
  note={arXiv:2412.13519}
}

• arXiv: 2412.13519