Scireplicate Bench Eval

scireplicate-bench-eval

SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers — Xiang et al. (2025) (arXiv:2504.00255, 2025)

What this evaluates

Evaluates an LLM's ability to comprehend algorithmic descriptions from academic papers and translate them into executable code. It probes the model's capacity for algorithmic reasoning, dependency resolution, and practical implementation within a repository context.

Datasets

• SciReplicate-Bench — total 100; splits: test (100); repo https://github.com/xyzCS/SciReplicate-Bench

Metrics

• Execution Accuracy (primary) — range: [0, 1]
  • Proportion of tasks where the generated code executes successfully without syntax or runtime errors. Calculated as the number of successfully executing tasks divided by the total number of tasks.
• Reasoning Graph Accuracy — range: [0, 1]
  • Node-level matching accuracy between the model's inferred algorithmic reasoning graph and the ground-truth graph. Evaluated using GPT-4o with temperature=0, top_p=1, averaged over three runs.
• CodeBLEU — range: [0, 1]
  • Code similarity metric combining n-gram precision, weighted n-gram precision, syntax matching, and data-flow matching between generated and reference code.
• Dependency/API Recall — range: [0, 1]
  • Recall scores for intra-file dependencies, cross-file dependencies, and external API usage identified in the generated code compared to the reference implementation.

Input / output format

Input: Algorithm description and function signature extracted from a research paper, optionally augmented with repository code context and paper text when using agent frameworks.

Output: Generated source code implementing the specified algorithm/function.

Scoring recipe

def compute_exe_acc(predictions, gold_tasks):
    success_count = 0
    for pred_code, task in zip(predictions, gold_tasks):
        try:
            # Execute in isolated sandbox
            exec(pred_code, {"__builtins__": {}})
            success_count += 1
        except Exception:
            pass
    return success_count / len(predictions)

Common pitfalls

• Missing or mismatched implementation details in paper descriptions (e.g., hyperparameters, numerical stability tricks) cause false execution failures.
• Reasoning Graph Accuracy relies on GPT-4o for node matching, introducing randomness; results must be averaged over three runs with temperature=0 to be reproducible.
• Overthinking in reasoning LLMs leads to excessive internal deliberation and reduced tool usage, artificially lowering execution accuracy compared to non-reasoning models.

Evidence (verbatim from paper)

Most LLMs perform poorly, achieving less than 0.1 execution accuracy without using the agent to examine literature and repository contexts.

Citation

@misc{xiang2025scireplicatebench,
  title={SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers},
  author={Xiang et al. (2025)},
  year={2025},
  note={arXiv:2504.00255}
}

• arXiv: 2504.00255