Wildsci Eval

wildsci-eval

WildSci: Advancing Scientific Reasoning from In-the-Wild Literature — Liu et al. (2026) (arXiv:2601.05567, 2026)

What this evaluates

Evaluates language models' scientific reasoning capabilities by testing their ability to answer domain-specific multiple-choice questions derived from peer-reviewed literature and established scientific benchmarks.

Datasets

• WildSci-Val — total 900; splits: val (900)
• GPQA-Aug — total 792; splits: test (792)
• SuperGPQA — total 26529; splits: test (26529)
• MMLU-Pro — total 12032; splits: test (12032)

Metrics

• accuracy (primary) — range: [0, 1]
  • Calculated as the proportion of questions where the model's final selected option exactly matches the ground-truth correct answer.

Input / output format

Input: Multiple-choice scientific questions with 4 or 10 options and a single correct answer.

Output: The model's final selected option (e.g., 'A', 'B', 'C', or 'D').

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for pred, gold_ans in zip(predictions, gold) if pred.strip().upper() == gold_ans.strip().upper())
    return correct / len(predictions)

Common pitfalls

• Answer choice order bias is explicitly mitigated for GPQA by using GPQA-Aug (4 permutations), but this augmentation is not applied to other datasets.
• Models often output chain-of-thought reasoning before the final answer; only the final selected option is scored, requiring robust parsing to ignore intermediate text.

Evidence (verbatim from paper)

We report accuracy for each dataset by checking whether the final selected option in the model’s response is correct.

Citation

@misc{liu2026wildsci,
  title={WildSci: Advancing Scientific Reasoning from In-the-Wild Literature},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2601.05567}
}

• arXiv: 2601.05567