Repro Bench Eval

repro-bench-eval

REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research? — Hu et al. (2025) (arXiv:2507.18901, 2025)

What this evaluates

Evaluates whether agentic AI systems can accurately assess the computational reproducibility of social science research by comparing original paper findings against results reproduced from provided raw data and code. It probes end-to-end agentic reasoning, including command execution, debugging, and result interpretation in a simulated research environment.

Datasets

• REPRO-Bench — total 112; splits: test (112); repo https://github.com/uiuc-kang-lab/REPRO-Bench

Metrics

• accuracy (primary) — range: [0, 1]
  • Binary metric measuring whether the agent's generated reproducibility score exactly matches the ground truth score. Calculated as the fraction of correctly scored instances out of the total.
• applicability_rate — range: [0, 1]
  • Measures whether the agent generates valid output files following instructions. Validity requires two conditions: (1) correct format and naming convention, and (2) placement in the designated root execution directory. Reports both original (only format checked) and adjusted (format + directory checked) rates.

Input / output format

Input: A working directory containing paper.pdf, a reproduction_package/ subdirectory, and a task prompt specifying the major findings to reproduce. The environment provides full CLI access with preinstalled software (Stata, MATLAB, LaTeX) and returns stdout/stderr feedback for executed commands.

Output: A reproducibility score and validity assessment saved to a correctly formatted and named output file placed in the agent's root execution directory.

Scoring recipe

def compute_accuracy(pred_score, gold_score):
    return 1.0 if pred_score == gold_score else 0.0

def compute_applicability(pred_file_path, pred_file_content):
    format_ok = check_format_and_naming(pred_file_content)
    dir_ok = pred_file_path == designated_root_dir
    return format_ok and dir_ok

# Adjusted applicability requires both format and directory compliance
# Original applicability only checks format compliance

Common pitfalls

• Agents may produce correctly formatted output files but save them outside the designated root directory, causing them to fail the adjusted applicability metric despite correct content.
• The $4 per-task API cost limit often terminates agents prematurely, artificially lowering reported accuracy and applicability rates compared to unconstrained runs.
• Evaluators must distinguish between original and adjusted metrics, as adjusted rates penalize directory placement errors even when output format is correct.

Evidence (verbatim from paper)

For performance evaluation, we use accuracy as the primary metric, measuring whether the generated reproducibility score matches the ground truth. We examine applicability rates to verify whether the agent generates valid outputs following the instructions in Section [3.4]. Validity is evaluated in two dimensions: the output file (1) must follow the correct format and naming convention, and (2) must be placed in the root directory where the agent starts executing.

Citation

@misc{hu2025reprobench,
  title={REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?},
  author={Hu et al. (2025)},
  year={2025},
  note={arXiv:2507.18901}
}

• arXiv: 2507.18901