bagel-eval
BAGEL: Benchmarking Animal Knowledge Expertise in Language Models — Shen et al. (2026) (arXiv:2604.16241, 2026)
What this evaluates
BAGEL probes language models' specialized knowledge of animal natural history, including taxonomy, morphology, behavior, habitat, vocalization, and ecological interactions. It evaluates closed-book fact recall and reasoning across diverse source domains (encyclopedic, scientific literature, ecological databases, and bioacoustics) without providing source passages at inference time.
Datasets
- BAGEL — total 11852; splits: test (11852)
Metrics
accuracy (primary) — range: [0, 1]
- Standard multiple-choice accuracy: the fraction of questions where the model's predicted option exactly matches the single correct answer.
Input / output format
Input: A unified prompt containing only the task instruction, the question stem, and four enumerated answer options.
Output: A single selected answer option from the four enumerated choices.
Scoring recipe
correct = 0
for pred, gold in zip(predictions, gold_answers):
if pred == gold:
correct += 1
accuracy = correct / len(predictions)
Common pitfalls
- Models are evaluated with deterministic decoding (seed 0, greedy), so results represent a single run rather than an average across seeds, which may introduce variance.
- Difficulty labels (easy/medium/hard) are reference strata based on agreement between GPT-5.4 and Claude Opus 4.6, not absolute difficulty, and hard items may contain option-level ambiguity.
- The benchmark is strictly closed-book; source passages used during construction are withheld at inference, preventing retrieval-based shortcuts.
Evidence (verbatim from paper)
We report accuracy on each source domain and an overall score across domains. All models are evaluated in a closed-book multiple-choice setting. At test time, the model is given a unified prompt containing only the task instruction, the question stem, and four enumerated answer options; the source passage used during benchmark construction is not provided at inference time. We use deterministic decoding with seed 0 and greedy generation for the reported results, so each model is represented by a single run rather than an average across seeds.
Citation
@misc{shen2026bagel,
title={BAGEL: Benchmarking Animal Knowledge Expertise in Language Models},
author={Shen et al. (2026)},
year={2026},
note={arXiv:2604.16241}
}
1---2name: bagel-eval3description: BAGEL probes language models' specialized knowledge of animal natural history, including taxonomy, morphology, behavior, habitat, vocalization, and ecological interactions. It evaluates closed-book fact recall and reasoning across diverse source domains (encyclopedic, scientific literature, ecological databases, and bioacoustics) without providing source passages at inference time. Use when the user wants to benchmark on BAGEL, or asks about evaluating this task. Reports accuracy.4---56# bagel-eval78> BAGEL: Benchmarking Animal Knowledge Expertise in Language Models — Shen et al. (2026) (arXiv:2604.16241, 2026)910## What this evaluates1112BAGEL probes language models' specialized knowledge of animal natural history, including taxonomy, morphology, behavior, habitat, vocalization, and ecological interactions. It evaluates closed-book fact recall and reasoning across diverse source domains (encyclopedic, scientific literature, ecological databases, and bioacoustics) without providing source passages at inference time.1314## Datasets1516- **BAGEL** — total 11852; splits: test (11852)1718## Metrics1920- `accuracy` **(primary)** — range: [0, 1]21 - Standard multiple-choice accuracy: the fraction of questions where the model's predicted option exactly matches the single correct answer.2223## Input / output format2425**Input**: A unified prompt containing only the task instruction, the question stem, and four enumerated answer options.2627**Output**: A single selected answer option from the four enumerated choices.2829## Scoring recipe3031```python32correct = 033for pred, gold in zip(predictions, gold_answers):34 if pred == gold:35 correct += 136accuracy = correct / len(predictions)37```3839## Common pitfalls4041- Models are evaluated with deterministic decoding (seed 0, greedy), so results represent a single run rather than an average across seeds, which may introduce variance.42- Difficulty labels (easy/medium/hard) are reference strata based on agreement between GPT-5.4 and Claude Opus 4.6, not absolute difficulty, and hard items may contain option-level ambiguity.43- The benchmark is strictly closed-book; source passages used during construction are withheld at inference, preventing retrieval-based shortcuts.4445## Evidence (verbatim from paper)4647> We report accuracy on each source domain and an overall score across domains. All models are evaluated in a closed-book multiple-choice setting. At test time, the model is given a unified prompt containing only the task instruction, the question stem, and four enumerated answer options; the source passage used during benchmark construction is not provided at inference time. We use deterministic decoding with seed 0 and greedy generation for the reported results, so each model is represented by a single run rather than an average across seeds.4849## Citation5051```bibtex52@misc{shen2026bagel,53 title={BAGEL: Benchmarking Animal Knowledge Expertise in Language Models},54 author={Shen et al. (2026)},55 year={2026},56 note={arXiv:2604.16241}57}58```5960- arXiv: 2604.16241
qhjqhj00