Protein Understanding Eval

protein-understanding-eval

xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein — Chen et al. (2024) (arXiv:2401.06199, 2024)

What this evaluates

Evaluates protein language models on sequence understanding across multiple downstream tasks (structure, function, interactions, developability) and 3D structure prediction from single amino acid sequences.

Datasets

• CAMEO — total 194; splits: test (194)
• CASP15 — total 56; splits: test (56)
• OOD Protein Sequences (UniProt) — total 20000; splits: test (10000), test (10000)

Metrics

• TM-score (primary) — range: [0, 1]
  • A structural similarity metric that measures the global fold similarity between predicted and experimental protein structures. Values range from 0 to 1, with higher scores indicating better structural alignment. Computed using standard superposition algorithms like TM-align.
• Perplexity — range: other
  • Quantifies the uncertainty of a language model in predicting a sequence. Calculated as the exponential of the average negative log-likelihood of the tokens. Lower values indicate better predictive accuracy.

Input / output format

Input: Single amino acid sequences (strings). For structure prediction, single and pair representations are extracted and fed into the structure module.

Output: For perplexity: token probability distributions. For structure prediction: 3D atomic coordinates. For downstream tasks: task-specific labels or regression values.

Scoring recipe

def compute_tm_score(pred_coords, true_coords):
    # Standard TM-align superposition and score calculation
    return tm_align_score(pred_coords, true_coords)

def compute_perplexity(logits, targets):
    log_probs = torch.nn.functional.log_softmax(logits, dim=-1)
    loss = -log_probs[range(len(targets)), targets].mean()
    return torch.exp(loss).item()

Common pitfalls

• Comparing PLM-based structure predictors (xT-Fold) directly with MSA-based models (AlphaFold2) without accounting for the fundamental difference in input requirements (single sequence vs. multiple sequence alignment).
• Assuming full fine-tuning is feasible for 100B-parameter models; the protocol explicitly uses LoRA to reduce trainable parameters to <1%.
• Confusing the two OOD datasets filtered at 0.5 and 0.9 sequence identity thresholds, which yield different perplexity baselines.

Evidence (verbatim from paper)

xT-Fold achieved a TM-score of 0.86 on the CAMEO dataset and 0.70 on CASP15. The scores for ESMFold were 0.85 and 0.65, respectively, while OmegaFold scored 0.80 and 0.60 on these datasets.

Citation

@misc{chen2024xtrimopglm,
  title={xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2401.06199}
}

• arXiv: 2401.06199