Protein Generative AI Eval

protein-generative-ai-eval

Generative Modeling in Protein Design: Neural Representations, Conditional Generation, and Evaluation Standards — Wanasekara et al. (2026) (arXiv:2603.26378, 2026)

What this evaluates

This evaluation framework probes the physical validity, functional success, and generalization capability of generative protein models. It emphasizes leakage-aware dataset splits, structural and docking quality metrics, and experimental validation pipelines to ensure designs are biologically plausible and functionally active.

Datasets

• PLINDER — total ?; splits: (unstated)
• PoseBusters — total ?; splits: (unstated)
• PDFBench — total ?; splits: (unstated)
• PoseX — total ?; splits: (unstated)
• VenusX — total ?; splits: (unstated)
• FragBench — total ?; splits: (unstated)
• GeomMotif — total ?; splits: (unstated)

Metrics

• RMSD (primary) — range: other
  • Root mean square deviation between predicted and reference atomic coordinates. Used for structure prediction and docking pose validity.
• scRMSD — range: other
  • Folds the designed sequence with a structure predictor and compares the predicted structure to the intended backbone to assess de novo design self-consistency.
• sequence recovery — range: percent
  • Fraction of positions matching the native sequence on a fixed backbone, standard for inverse folding.
• pairwise sequence dissimilarity — range: other
  • Avg. Dissimilarity = (2/(k(k-1))) * sum_{i<j} d(s_i, s_j), where d is Hamming or Levenshtein distance over k generated sequences.
• Shannon entropy — range: other
  • H = -sum_{a in A} p_a log_2 p_a, measuring amino acid variability at each position to distinguish conserved cores from flexible surfaces.
• DockQ — range: [0, 1]
  • Interface-aware metric reflecting protein-protein interaction quality, preferred over global backbone RMSD for complexes.
• PoseBusters validity — range: percent
  • Checks physical plausibility constraints including steric clash rates, chirality, and valence consistency for docking poses.

Input / output format

Input: Conditioned protein sequences, backbones, or complexes; reference structures/sequences for ground-truth comparison.

Output: Generated protein structures (backbones), sequences, or complexes; confidence scores (e.g., pLDDT, pTM); validity flags and metric scores.

Scoring recipe

def evaluate(predictions, gold, k=None):
    # Structural similarity
    rmsd = np.sqrt(np.mean((predictions.coords - gold.coords)**2))
    # De novo self-consistency
    sc_rmsd = fold_and_compare_rmsd(predictions.seq, gold.backbone)
    # Sequence recovery
    recovery = np.mean(predictions.seq == gold.seq)
    # Pairwise dissimilarity
    dissim = 0
    for i in range(k):
        for j in range(i+1, k):
            dissim += hamming_distance(seq[i], seq[j])
    dissim /= (k * (k - 1) / 2)
    # Physical validity
    validity = check_steric_clashes(predictions) and check_valence(predictions)
    return rmsd, sc_rmsd, recovery, dissim, validity

Common pitfalls

• Random splits cause severe information leakage due to near-duplicates at sequence, structure, or ligand-scaffold levels.
• Relying solely on geometric similarity (RMSD) misrepresents functional success or physical plausibility.
• Computational metrics are only proxies; credible claims require wet-lab experimental validation.

Evidence (verbatim from paper)

For inverse folding, sequence recovery (fraction of positions matching the native sequence on a fixed backbone) is the standard benchmark. Pairwise sequence dissimilarity quantifies diversity among generated sequences: Avg. Dissimilarity = (2/(k(k-1))) sum_{i<j} d(s_i, s_j), where d is Hamming or Levenshtein distance over a set of k generated sequences.

Citation

@misc{wanasekara2026generative,
  title={Generative Modeling in Protein Design: Neural Representations, Conditional Generation, and Evaluation Standards},
  author={Wanasekara et al. (2026)},
  year={2026},
  note={arXiv:2603.26378}
}

• arXiv: 2603.26378