Dbench Bio Eval

dbench-bio-eval

Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery — Chaoqun Yang et al. (2026) (arXiv:2603.03322, 2026)

What this evaluates

Evaluates whether large language models can discover genuinely new biological knowledge by generating correct scientific hypotheses or mechanisms from post-release literature, enforcing strict temporal separation to prevent data leakage.

Datasets

• DBench-Bio — total ?; splits: test (-1); repo https://github.com/transcend-0/DBench-Bio

Metrics

• Score (primary) — range: [0, 5]
  • A 5-point evaluation score assessing the correctness and relevance of the model's generated biological mechanism or discovery answer against the ground truth. Higher scores indicate more accurate and specific hypothesis generation.

Input / output format

Input: A question or prompt asking the model to derive a specific biological mechanism or discovery based on recent, post-release scientific literature.

Output: A textual answer describing the proposed biological mechanism or discovery.

Scoring recipe

def compute_score(predictions, gold):
    # Rubric inferred from text: 0-5 scale based on correctness/specificity
    if prediction is refusal or hallucination:
        return 0
    elif prediction matches generic textbook knowledge:
        return 1
    elif prediction partially matches ground truth mechanism:
        return 3
    elif prediction fully matches ground truth with specific details:
        return 5
    else:
        return 2

Common pitfalls

• Data leakage is mitigated by temporal separation, but models may still rely on pre-training data that overlaps with retrieved tool outputs.
• High performance on static knowledge benchmarks (e.g., MMLU-Pro) does not correlate with discovery ability, making cross-benchmark comparisons misleading.
• Tool-use (RAG/ReAct) often retrieves information already present in the model's internal weights, yielding negligible gains unless the model's reasoning capacity is strong.

Evidence (verbatim from paper)

The aggregate performance across all evaluated models remains low (The maximum Score is 5). This underscores the inherent difficulty of knowledge discovery, suggesting that current LLM models have yet to master the ability to derive new knowledge.

Citation

@misc{yang2026dbenchbio,
  title={Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery},
  author={Chaoqun Yang et al. (2026)},
  year={2026},
  note={arXiv:2603.03322}
}

• arXiv: 2603.03322