Psp Protein Structure Eval

psp-protein-structure-eval

PSP: Million-level Protein Sequence Dataset for Protein Structure Prediction — Liu et al. (2022) (arXiv:2206.12240, 2022)

What this evaluates

Evaluates protein structure prediction models on their ability to infer accurate 3D atomic coordinates from amino acid sequences. It specifically probes topological backbone similarity and side-chain accuracy when trained on large-scale distilled protein datasets.

Datasets

• CASP14 test set — total 87; splits: test (87)
• PSP dataset — total 1315000; splits: train (-1)

Metrics

• TM-score (primary) — range: [0, 1]
  • Assesses topological similarity between the predicted and true protein structures. A score of 1 indicates an exact match, while scores below 0.17 indicate irrelevant structures.
• lDDT — range: [0, 1]
  • local Distance Difference Test measuring side-chain atom position accuracy. A score of 0 indicates totally different structures, and 1 indicates exactly the same.

Input / output format

Input: Amino acid sequences accompanied by multiple sequence alignments (MSA) and template information.

Output: Predicted 3D protein structure represented as atomic coordinates (backbone and side-chain atoms).

Scoring recipe

def evaluate_structure(pred_coords, true_coords):
    # Calculate topological similarity for backbone
    tmscore = calculate_tmscore(pred_coords, true_coords)
    # Calculate side-chain atom position accuracy
    lddt = calculate_lddt(pred_coords, true_coords)
    return tmscore, lddt

# Protocol: Average TM-score over 87 CASP14 sequences
avg_tmscore = mean([evaluate_structure(p, t)[0] for p, t in casp14_pairs])

Common pitfalls

• Data leakage must be prevented by strictly excluding CASP14 sequences with structure release dates after May 13, 2020.
• Dropout causes training instability and rapid TM-score drops during fine-tuning; it must be disabled.
• The violation loss clash term can cause training crashes due to high variance on distillation data; it requires clipping (e.g., to 5.0).

Evidence (verbatim from paper)

We took the TM-score on 87 CASP14 sequences as the monitor of accuracy. The TM-score (template modeling score) is used to assess the topological similarity between protein target structure and predicted structure. A value 1 means exactly the same, and <0.17 means totally irrelevant.

Citation

@misc{liu2022psp,
  title={PSP: Million-level Protein Sequence Dataset for Protein Structure Prediction},
  author={Liu et al. (2022)},
  year={2022},
  note={arXiv:2206.12240}
}

• arXiv: 2206.12240