Scibench Eval

scibench-eval

SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models — Wang et al. (2023) (arXiv:2307.10635, 2023)

What this evaluates

This benchmark evaluates large language models' ability to solve college-level scientific problems across mathematics, chemistry, and physics. It probes multi-step quantitative reasoning, unit conversion, physical derivations, and the efficacy of prompting strategies and external computational tools.

Datasets

• SciBench — total ?; splits: textbook (-1), multimodal (-1), exam (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of problems solved correctly. For textbook problems, a prediction is correct if it matches the ground truth answer within a 5% relative tolerance. For the exam subset, solutions are graded using instructor-provided rubrics rather than exact matching.

Input / output format

Input: Textual problem statements from college-level science textbooks, optionally accompanied by visual contexts (for the multimodal subset). Prompts include a system prompt, zero- or few-shot examples, and instructions for Chain-of-Thought reasoning or Python tool use.

Output: Natural language reasoning steps and final numerical/textual answer. In tool-augmented settings, the model outputs Python code intended to compute the solution.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_answers):
    if subset == 'exam':
        score = grade_with_rubric(pred, gold)
    else:
        try:
            pred_val = extract_number(pred)
            if abs(pred_val - gold) / abs(gold) <= 0.05:
                score = 1.0
            else:
                score = 0.0
        except:
            score = 0.0
    correct += score
accuracy = (correct / len(predictions)) * 100

Common pitfalls

• The 5% relative tolerance applies to textbook problems, not absolute difference; failing to implement relative tolerance will skew results.
• The exam subset requires rubric-based grading rather than simple answer matching, as stated in the setup.
• Tool-augmented (Python/Wolfram) approaches were only evaluated in the few-shot setting, not zero-shot.
• All experiments used a temperature of 0 to minimize randomness, which must be replicated for fair comparison.

Evidence (verbatim from paper)

In terms of evaluation metrics, we compare the model outputs with the correct answers, allowing a relative tolerance of 5%. In particular to the exam dataset, the model solutions are graded using the rubrics provided by the instructors. We report the model performance in terms of accuracy score for each textbook and an average score over all problems.

Citation

@misc{wang2023scibench,
  title={SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2307.10635}
}

• arXiv: 2307.10635