Nucleotide Transformer Benchmark Eval

nucleotide-transformer-benchmark-eval

JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model — Duan et al. (2025) (arXiv:2505.17257, 2025)

What this evaluates

Assesses a model's capability to predict various genomic features including histone markers, regulatory annotations, and splice sites. It tests fine-grained sequence understanding and multi-task classification across diverse genomic contexts.

Datasets

• Nucleotide Transformer Benchmark — total ?; splits: 10-fold CV (-1)

Metrics

• MCC — range: [-1, 1]
  • Matthews Correlation Coefficient, measuring the quality of binary classifications.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall.
• accuracy (primary) — range: [0, 1]
  • Fraction of correctly classified sequences.

Input / output format

Input: DNA sequences for histone marker prediction, regulatory annotation, and splice site annotation tasks.

Output: Class labels for each specific genomic feature task.

Scoring recipe

if task in ['histone', 'enhancer']:
    score = mcc(gold, pred)
elif task in ['promoter', 'splice_acceptor', 'splice_donor']:
    score = f1_score(gold, pred)
else:
    score = accuracy(gold, pred)

Common pitfalls

• Metrics vary by task type; using a single metric across all tasks would misrepresent performance.
• The benchmark includes tasks with highly imbalanced classes, making accuracy misleading without MCC or F1.
• Results are averaged over 10-fold CV; error bars represent max-min across 10 seeds.

Evidence (verbatim from paper)

Next, we evaluate our model on the Nucleotide Transformer tasks, which include 18 datasets covering histone marker prediction, regulatory annotation prediction, and splice site annotation prediction. Following the evaluation metrics outlined in [nucleotidebenchmark], we perform 10-fold cross-validation for each task... Metrics vary by task: MCC for histone markers and enhancer annotation, F1-score for promoter annotation and splice site acceptor/donor, and accuracy for splice site “all”.

Citation

@misc{duan2025janusdna,
  title={JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model},
  author={Duan et al. (2025)},
  year={2025},
  note={arXiv:2505.17257}
}

• arXiv: 2505.17257