Deepprotein Benchmark Eval

deepprotein-benchmark-eval

DeepProtein: Deep Learning Library and Benchmark for Protein Sequence Learning — Jiaqing Xie et al. (2024) (arXiv:2410.02023, 2024)

What this evaluates

Evaluates deep learning models on a comprehensive suite of protein sequence learning tasks, including function prediction, subcellular localization, protein-protein interaction, epitope/paratope prediction, antibody developability, CRISPR repair outcomes, and protein structure prediction.

Datasets

• Fluorescence — total 54025; splits: train (-1), val (-1), test (-1)
• Stability — total 68934; splits: train (-1), val (-1), test (-1)
• β-lactamase — total 5198; splits: train (-1), val (-1), test (-1)
• Solubility — total 71419; splits: train (-1), val (-1), test (-1)
• Subcellular — total 13961; splits: train (-1), val (-1), test (-1)
• Binary — total 8634; splits: train (-1), val (-1), test (-1)
• PPI Affinity — total 2682; splits: train (-1), val (-1), test (-1)
• Yeast — total 2172; splits: train (-1), val (-1), test (-1)
• Human PPI — total 7348; splits: train (-1), val (-1), test (-1)
• IEDB — total 3159; splits: train (-1), val (-1), test (-1)
• PDB-Jespersen — total 447; splits: train (-1), val (-1), test (-1)
• SAbDab-Liberis — total 1023; splits: train (-1), val (-1), test (-1)
• TAP — total 242; splits: train (-1), val (-1), test (-1)
• SAbDab-Chen — total 2409; splits: train (-1), val (-1), test (-1)
• CRISPR-Leenay — total 1521; splits: train (-1), val (-1), test (-1)
• Fold — total 13766; splits: train (-1), val (-1), test (-1)
• Secondary Structure — total 11361; splits: train (-1), val (-1), test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted labels over total samples. Used for classification tasks (e.g., Solubility, Subcellular, PPI, Fold).
• MSE — range: [0, inf)
  • Mean Squared Error between predicted and ground-truth continuous values. Used for regression tasks (e.g., Fluorescence, Stability, β-lactamase, PPI Affinity, TAP, SAbDab-Chen).

Input / output format

Input: Amino acid sequences (single or paired for PPI/developability), optionally augmented with structural features or edge information for graph-based models.

Output: Predicted class labels, binary indicators, or continuous values depending on the task.

Scoring recipe

def compute_metric(predictions, gold, task_type):
    if task_type == 'classification':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task_type == 'regression':
        return sum((p - g) ** 2 for p, g in zip(predictions, gold)) / len(gold)
    elif task_type == 'residue_level':
        return residue_level_aggregate(predictions, gold)
    return None

Common pitfalls

• Splits strictly follow PEER benchmark and TDC conventions rather than random splits; using random splits causes data leakage and invalidates results.
• Residue-level tasks (epitope, paratope, secondary structure) require sequence-to-sequence alignment and residue-level aggregation, not sequence-level averaging.
• Paired-input tasks (PPI, developability) must maintain correct sequence pairing during batching and evaluation to avoid mismatched labels.

Evidence (verbatim from paper)

In this library, we follow the train-validation-test split in PEER benchmark (Xu et al., 2022) and TDC (Huang et al., 2022). Each individual split is reported from Table 2 to 7.

Citation

@misc{xie2024deepprotein,
  title={DeepProtein: Deep Learning Library and Benchmark for Protein Sequence Learning},
  author={Jiaqing Xie et al. (2024)},
  year={2024},
  note={arXiv:2410.02023}
}

• arXiv: 2410.02023