Omnigenbench Eval

omnigenbench-eval

OmniGenBench: Automating Large-scale in-silico Benchmarking for Genomic Foundation Models — Yang et al. (2024) (arXiv:2410.01784, 2024)

What this evaluates

Evaluates genomic foundation models on diverse in-silico tasks including RNA structure prediction, plant DNA regulation, cross-species genomic understanding, and regulatory element classification. It probes the models' ability to generalize across nucleic acid types, species, and complex sequence motifs.

Datasets

• RGB — total ?; splits: test (-1); repo https://github.com/yangheng95/OmniGenomeBench
• PGB — total ?; splits: test (-1); repo https://github.com/yangheng95/OmniGenomeBench
• GUE — total ?; splits: test (-1); repo https://github.com/yangheng95/OmniGenomeBench
• GB — total ?; splits: test (-1); repo https://github.com/yangheng95/OmniGenomeBench

Metrics

• macro F1 (primary) — range: [0, 1]
  • The unweighted mean of recall computed for each class. Calculated as the average of per-class F1 scores to handle class imbalance in multi-class classification tasks.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to distinguish between classes across all classification thresholds.
• RMSE — range: other
  • Root Mean Squared Error. The square root of the average of squared differences between predicted and actual continuous values.
• Accuracy — range: [0, 1]
  • The ratio of correctly predicted instances to the total number of instances.

Input / output format

Input: Nucleotide sequences (RNA or DNA) provided as input strings, often up to 6,000 bases, paired with task-specific labels (e.g., structural states, continuous values, or regulatory class labels).

Output: Model predictions in the format required by the task: continuous values for regression (e.g., degradation rate, accessibility), class labels or probabilities for classification (e.g., splice sites, enhancer regions), or structural predictions for RNA folding.

Scoring recipe

def evaluate(predictions, gold, task_type):
    if task_type == 'classification':
        return macro_f1_score(gold, predictions)
    elif task_type == 'regression':
        return rmse(gold, predictions)
    elif task_type == 'binary':
        return roc_auc_score(gold, predictions)
    else:
        return accuracy_score(gold, predictions)

Common pitfalls

• Using standard accuracy instead of macro F1 for imbalanced classification tasks, which skews results toward majority classes.
• Assuming RNA-trained models will perform equally well on DNA tasks without accounting for modality differences and sequence motif variations.
• Overlooking cross-species generalization gaps, as models often excel on training species but fail on others due to divergent genomic architectures.

Evidence (verbatim from paper)

To mitigate potential class imbalance issues, we adopt the macro F1 score as the metric in classification tasks, replacing accuracy where necessary.

Citation

@misc{yang2024omnigenbench,
  title={OmniGenBench: Automating Large-scale in-silico Benchmarking for Genomic Foundation Models},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2410.01784}
}

• arXiv: 2410.01784