Reproducibility Repair Eval

reproducibility-repair-eval

Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches — Shah et al. (2026) (arXiv:2602.08561, 2026)

What this evaluates

Evaluates the ability of LLMs and AI agents to automatically repair broken R-based social science code and restore computational reproducibility. It probes how well different workflows handle varying error complexities and contextual information.

Datasets

• Custom R-based Social Science Code Dataset — total ?; splits: test (-1); repo https://github.com/Mehtab07/Automating-Computational-Reproducibility

Metrics

• reproduction_success_rate (primary) — range: percent
  • Percentage of test cases where the repaired script executes successfully and produces output identical to the original study's ground truth results. Calculated as (successful reproductions / total cases) × 100.

Input / output format

Input: For prompt-based: R script with injected errors, error logs, and context prompts (minimal, medium, or full including original paper/scripts). For agent-based: Same code + containerized execution environment allowing iterative inspection, modification, and re-execution.

Output: Repaired R script intended to restore original analytical outputs.

Scoring recipe

def calculate_success_rate(predictions, gold_outputs):
    successful = 0
    for pred, gold in zip(predictions, gold_outputs):
        result = execute_script_in_isolated_env(pred)
        if result is not None and result == gold:
            successful += 1
    return (successful / len(predictions)) * 100

Common pitfalls

• Success is strictly defined by exact output reproduction matching the original study, not just successful code execution.
• Performance is highly sensitive to prompt context richness; adding irrelevant or excessive context can sometimes degrade LLM performance.
• Error complexity is categorized into three levels (A: straightforward, B: missing libraries/small code, C: complex logic), which drastically affect success rates.

Evidence (verbatim from paper)

We evaluated performance using the same metrics across workflows, focusing on reproduction success rates broken down by error category, prompt type, model or agent used, and reproducibility status as determined by output validation.

Citation

@misc{shah2026automating,
  title={Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches},
  author={Shah et al. (2026)},
  year={2026},
  note={arXiv:2602.08561}
}

• arXiv: 2602.08561