Dna Foundation Eval

dna-foundation-eval

BMFM-DNA: A SNP-aware DNA foundation model to capture variant effects — Li et al. (2025) (arXiv:2507.05265, 2025)

What this evaluates

Evaluates genomic foundation models on multiple biological prediction tasks, including regulatory element detection, splicing, and variant-disease association, to measure their ability to capture functional DNA sequences and SNP effects.

Datasets

• Promoter detection — total ?; splits: test (-1)
• Core promoter detection — total ?; splits: test (-1)
• TF binding detection — total ?; splits: test (-1)
• Splicing detection — total ?; splits: test (-1)
• lenti-MPRA K562 — total ?; splits: test (-1)
• SNP-to-disease association — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used for binary/multi-class detection tasks.
• AUC — range: [0, 1]
  • Area under the receiver operating characteristic curve, measuring the model's ability to distinguish between classes across all classification thresholds.
• PCC — range: [-1, 1]
  • Pearson Correlation Coefficient measuring the linear correlation between predicted and observed continuous values.
• MCC — range: [-1, 1]
  • Matthews Correlation Coefficient, a balanced measure of binary classification quality that accounts for true/false positives and negatives.

Input / output format

Input: DNA sequences (with optional SNP/variant annotations) tokenized via BPE.

Output: Task-specific predictions: binary/multi-class labels for detection tasks, continuous values for lenti-MPRA, and disease association scores for SNP-to-disease.

Scoring recipe

def compute_metrics(predictions, labels, task_type):
    if task_type in ['Promoter', 'cPromoter', 'TF binding', 'Splicing']:
        precision = tp / (tp + fp + 1e-8)
        recall = tp / (tp + fn + 1e-8)
        f1 = 2 * precision * recall / (precision + recall + 1e-8)
        return f1
    elif task_type == 'SNP-to-disease':
        return roc_auc_score(labels, predictions)
    elif task_type == 'lenti-MPRA':
        return pearsonr(labels, predictions)[0]
    return None

Common pitfalls

• Different tasks report different metrics (F1, PCC, AUC), so cross-task comparisons must account for metric scaling and task type.
• Negative sample generation strategies heavily influence results; random SNP imputation (Class 2) artificially inflates F1, while randomized SNPs in positives (Class 4) creates a harder benchmark.
• Baselines like DNABERT-2 are pre-trained on 135 species genomes, making direct performance comparisons with human-only variant models potentially unfair.

Evidence (verbatim from paper)

To evaluate model performance, we use $F_{1}$ score, Matthews Correlation Coefficient (MCC), and area under the receiver operating characteristic curve (AUC). For each model, we train with three different random seeds and report the average performance. The overall performance of each model on specific tasks is summarized in Table 2.

Citation

@misc{li2025bmfmdna,
  title={BMFM-DNA: A SNP-aware DNA foundation model to capture variant effects},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2507.05265}
}

• arXiv: 2507.05265