Hybridna Eval

hybridna-eval

HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Model — Mingqian Ma et al. (2025) (arXiv:2502.10807, 2025)

What this evaluates

Evaluates the capability of DNA foundation models to perform short-range and long-range genomic understanding tasks, as well as their ability to generate biologically plausible cis-regulatory elements. It probes sequence classification, variant effect prediction, and generative design across multiple species and cell types.

Datasets

• GUE — total 28; splits: test (-1)
• BEND — total 3; splits: test (-1)
• LRB — total 2; splits: test (-1)
• CRE (regLM) — total 670000; splits: train (670000), test (-1)

Metrics

• MCC (primary) — range: [-1, 1]
  • Matthews Correlation Coefficient; measures the quality of binary/multiclass classifications considering true/false positives and negatives.
• F-1 score — range: [0, 1]
  • Harmonic mean of precision and recall; used specifically for the Covid Variant Classification task.
• AUROC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve; measures the ability of the model to distinguish between classes across all classification thresholds.
• AUPRC — range: [0, 1]
  • Area Under the Precision-Recall Curve; evaluates performance on imbalanced datasets by focusing on positive class predictions.
• Top-1 activity — range: other
  • The highest predicted enhancer activity score among the generated sequences for a given cell type.
• Mean activity — range: other
  • The average of the top 100 predicted enhancer activity scores for a given cell type.
• Diversity — range: other
  • The mean of pair-wise edit distances of the top 100 predicted sequences, measuring the overall diversity of high-quality generated sequences.

Input / output format

Input: DNA sequences (70–131,072 bp) with task-specific prompts or labels. For generation tasks, prompt tokens specifying cell type and activity strength (e.g., 3-digit labels for enhancer activity).

Output: Class probabilities or predicted labels for classification tasks; generated DNA sequences (200 bp) for CRE tasks.

Scoring recipe

# Classification
mcc = matthews_corrcoef(y_true, y_pred)
f1 = f1_score(y_true, y_pred, average='macro')
auroc = roc_auc_score(y_true, y_prob)
auprc = average_precision_score(y_true, y_prob)
# Generation
seqs = beam_search_decode(model, prompts, num=600)
scores = external_scoring_model.predict(seqs)
top1 = max(scores)
mean_act = mean(scores[:100])
diversity = mean(pairwise_edit_distance(s1, s2) for s1, s2 in combinations(seqs[:100], 2))

Common pitfalls

• Decoder-only models require specific embedding extraction strategies (last token hidden state vs. mean sequence hidden state) which drastically changes performance and must match the benchmark's original setting.
• Fine-tuning protocols (learning rates, epochs, frozen vs. full parameter training) must strictly follow the original benchmark settings to ensure fair comparison.
• Zero-shot vs. fine-tune evaluation modes for LRB tasks are often confused, leading to incorrect metric reporting.

Evidence (verbatim from paper)

Following the same setting as DNABERT-2, we use metrics of Matthews Correlation Coefficient (MCC) for all tasks, except for the Covid task, where we use the F-1 score according to the GUE dataset’s original setting.

Citation

@misc{ma2025hybridna,
  title={HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Model},
  author={Mingqian Ma et al. (2025)},
  year={2025},
  note={arXiv:2502.10807}
}

• arXiv: 2502.10807