Peta Protein Eval

peta-protein-eval

PETA: Evaluating the Impact of Protein Transfer Learning with Sub-word Tokenization on Downstream Applications — Yang Tan et al. (2023) (arXiv:2310.17415, 2023)

What this evaluates

Evaluates protein language models across 33 downstream tasks (fitness, localization, PPI, solubility, and structure prediction) to measure how sub-word tokenization and vocabulary size affect representation quality and task performance.

Datasets

• PETA Benchmark Suite — total ?; splits: test (-1); repo https://github.com/ginnm/ProteinPretraining

Metrics

• Spearman correlation (primary) — range: [-1, 1]
  • Measures the monotonic relationship between predicted and observed values. Computed as the rank correlation coefficient between model predictions and ground truth labels.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances.
• MSE — range: [0, inf)
  • Mean Squared Error between predicted and actual continuous values.

Input / output format

Input: Protein sequences (amino acid strings) tokenized via per-amino-acid, BPE, or Unigram strategies, constrained to a maximum sequence length of 1024 tokens.

Output: Continuous regression values or discrete class labels, depending on the downstream task.

Scoring recipe

def compute_metric(predictions, gold, task_type):
    if task_type == 'correlation':
        return spearmanr(predictions, gold).correlation
    elif task_type == 'accuracy':
        preds_class = np.argmax(predictions, axis=1)
        return np.mean(preds_class == gold)
    elif task_type == 'mse':
        return np.mean((predictions - gold) ** 2)

Common pitfalls

• Averaging results across multiple data splits, tokenization methods, classification heads, and random seeds can mask per-setting performance variations.
• Vocabulary sizes exceeding 800 tokens consistently degrade performance on structure prediction tasks, contrary to typical NLP scaling trends.
• Several datasets (e.g., GB1, AAV, Thermo, Solmut) contain multiple data partitioning methods that must be explicitly averaged before reporting final scores.

Evidence (verbatim from paper)

Fitness Prediction. Table 5 showcases results for five distinct tasks under the umbrella of Fitness Prediction and the evaluation metrics is Spearman correlation.

Citation

@misc{tan2023peta,
  title={PETA: Evaluating the Impact of Protein Transfer Learning with Sub-word Tokenization on Downstream Applications},
  author={Yang Tan et al. (2023)},
  year={2023},
  note={arXiv:2310.17415}
}

• arXiv: 2310.17415