Scicode Eval

scicode-eval

SciCode: A Research Coding Benchmark Curated by Scientists — Tian et al. (2024) (arXiv:2407.13168, 2024)

What this evaluates

Probes large language models' ability to perform scientific reasoning, domain-specific knowledge recall, and code synthesis on real-world research problems. It evaluates performance on both decomposed subproblems and full main problems under varying conditions of background knowledge and context carry-over.

Datasets

• SciCode — total 80; splits: test (80)

Metrics

• pass@1 (primary) — range: percent
  • pass@1 = (number of problems solved correctly on the first attempt / total number of problems) × 100. It measures the percentage of problems where the model's initial generated code passes all validation test cases.

Input / output format

Input: Problem description (main problem or subproblem), optional gold scientific background text, and optionally generated or gold solutions from previous subproblems.

Output: Executable code snippet solving the specified problem.

Scoring recipe

correct = 0
for problem in dataset:
    code = model.generate(problem.prompt)
    if execute_and_pass_tests(code, problem.test_cases):
        correct += 1
return (correct / len(dataset)) * 100

Common pitfalls

• Confusing the standard setup (no background knowledge, no carry-over) with the background-knowledge setup, which yields significantly higher scores.
• Assuming high subproblem pass@1 rates translate to main problem success; the gap is large due to cumulative error and instruction-following demands.
• Ignoring the requirement to carry over generated solutions from previous subproblems in the standard setup, which tests context management and instruction following.

Evidence (verbatim from paper)

Table 2 presents results under the standard setup.444Without background and carrying over generated subproblem solutions. See §[2.4] for a more detailed discussion. For the easier subproblem-level evaluation, the state-of-the-art models we test solve 14%-26% of the subproblems. Among them, Claude3.5-Sonnet achieves the best performance, with a 26.0% pass@1 rate.

Citation

@misc{tian2024scicode,
  title={SciCode: A Research Coding Benchmark Curated by Scientists},
  author={Tian et al. (2024)},
  year={2024},
  note={arXiv:2407.13168}
}

• arXiv: 2407.13168