Hyperhelm Eval

hyperhelm-eval

HyperHELM: Hyperbolic Hierarchy Encoding for mRNA Language Modeling — Max van Spengler et al. (arXiv:2509.24655, 2025)

What this evaluates

Evaluates the ability of mRNA language models to predict diverse biological properties (e.g., protein expression, degradation, thermostability) and annotate antibody sequence regions. It also probes model robustness to out-of-distribution sequence lengths and extreme GC content, testing generalization in hierarchical biological representation learning.

Datasets

• Ab1 — total 662; splits: train (-1), val (-1), test (-1)
• Ab2 — total 2672; splits: train (-1), val (-1), test (-1)
• mRFP — total 1459; splits: train (-1), val (-1), test (-1)
• COVID-19 Vaccine — total 2400; splits: train (-1), val (-1), test (-1)
• Drosophila melanogaster — total 10338; splits: train (-1), val (-1), test (-1)
• Saccharomyces cerevisiae — total 4937; splits: train (-1), val (-1), test (-1)
• Pichia pastoris — total 4682; splits: train (-1), val (-1), test (-1)
• Fungal — total 7056; splits: train (-1), val (-1), test (-1)
• E. coli — total 6348; splits: train (-1), val (-1), test (-1)
• iCodon — total 65357; splits: train (-1), val (-1), test (-1)
• Antibody Region Annotation — total 2000; splits: test (2000)

Metrics

• Spearman rank correlation (primary) — range: [-1, 1]
  • Measures the monotonic relationship between predicted and true continuous values. Computed as the Pearson correlation between the rank-transformed predictions and gold labels.
• accuracy — range: [0, 1]
  • Proportion of correctly predicted class labels out of the total number of instances.

Input / output format

Input: Codon-level tokenized mRNA sequences.

Output: Continuous property values for regression tasks, or discrete class labels (e.g., protein expression levels, antibody region types) for classification tasks.

Scoring recipe

def compute_spearman(pred, gold):
    return scipy.stats.spearmanr(pred, gold).correlation

def compute_accuracy(pred, gold):
    return sum(p == g for p, g in zip(pred, gold)) / len(gold)

Common pitfalls

• Missing values in reported results occur when models fail to process sequences due to length limitations, which can skew average performance comparisons.
• The paper uses predefined train/val/test splits rather than random splits; re-running requires obtaining these exact splits to ensure comparability.
• Regression tasks use Spearman rank correlation, while classification tasks (E. coli, antibody annotation) use accuracy; mixing these metrics will invalidate results.

Evidence (verbatim from paper)

Except for the E. coli classification task, all datasets provide regression labels for evaluating property prediction. Following prior works, we use predefined train/val/test data splits and report Spearman rank correlation for regression and accuracy for classification tasks.

Citation

@misc{vanspengler2025hyperhelm,
  title={HyperHELM: Hyperbolic Hierarchy Encoding for mRNA Language Modeling},
  author={Max van Spengler et al.},
  year={2025},
  note={arXiv:2509.24655}
}

• arXiv: 2509.24655