Gene Bench Eval

gene-bench-eval

Does your model understand genes? A benchmark of gene properties for biological and text models — Kan-Tor et al. (2024) (arXiv:2412.04075, 2024)

What this evaluates

This benchmark evaluates how well various deep learning models encode biological knowledge by predicting 312 ground-truth gene properties. It probes capabilities across five domains: genomic, regulatory, localization, biological processes, and protein features, using binary, multi-label, and multi-class classification tasks.

Datasets

• GeneBench — total 312; splits: test (-1); repo http://github.com/BiomedSciAI/gene-benchmark

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, measuring the probability that a randomly chosen positive instance ranks higher than a randomly chosen negative instance.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall, calculated as 2 * (precision * recall) / (precision + recall).

Input / output format

Input: Pre-computed gene embeddings (vectors) from text models, scRNA foundation models, DNA/protein sequence models, or classical baselines.

Output: Predicted probabilities or class labels for each of the 312 gene property tasks (binary, multi-label, or multi-class).

Scoring recipe

# For each of the 312 gene property tasks:
clf = LogisticRegression()
clf.fit(X_train, y_train)
y_prob = clf.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, y_prob)
# F1 is computed similarly using thresholded predictions
f1 = f1_score(y_test, clf.predict(X_test), average='weighted')

Common pitfalls

• Larger model size does not consistently yield better performance; smaller models often match or approach larger ones.
• Text-based models sometimes perform comparably to classical bag-of-words baselines, highlighting that raw embedding size isn't the sole driver of biological knowledge.
• Using non-linear models (e.g., MLPs) can introduce hyperparameter sensitivity; the authors recommend linear classifiers for robust cross-model comparison.

Evidence (verbatim from paper)

The performance of each model on the task families as measured by average area under the ROC curve. Parentheses show the corresponding standard deviation across all tasks of the same family. These trends are consistent when using other evaluation metrics such as F1 (see Figure[S1]).

Citation

@misc{kan-tor2024genebench,
  title={Does your model understand genes? A benchmark of gene properties for biological and text models},
  author={Kan-Tor et al. (2024)},
  year={2024},
  note={arXiv:2412.04075}
}

• arXiv: 2412.04075