Swe Bench Repair Eval

swe-bench-repair-eval

RepoRepair: Leveraging Code Documentation for Repository-Level Automated Program Repair — Zhongqiang Pan et al. (2026) (arXiv:2603.01048, 2026)

What this evaluates

This evaluation probes an LLM-based agent's ability to automatically locate faults and generate correct code patches for real-world software issues. It tests both traditional text-only bug fixing and multimodal reasoning where visual UI behavior must be understood alongside code.

Datasets

• SWE-bench Lite — total 323; splits: dev (23), test (300)
• SWE-bench Multimodal — total 619; splits: dev (102), test (517)

Metrics

• %Resolved (primary) — range: percent
  • Percentage of issues successfully fixed in the benchmark. All patches are validated via automated test suites; any patch that passes is considered correct.
• Avg. $Cost — range: other
  • Average inference cost per tool run.
• %Correct Localization — range: percent
  • Percentage of cases where the tool’s localized file set fully contains all developer-patched files. It is a recall-oriented metric.

Input / output format

Input: Issue description (text-only or multimodal with images/GIFs/videos), repository codebase, and retrieved context (top-50 relevant files, then top-5 suspicious files with functions/classes).

Output: A code patch (diff) intended to fix the reported issue, validated against the repository's test suite.

Scoring recipe

def compute_resolved_rate(predictions, test_suites):
    resolved = 0
    for issue_id, patch in predictions.items():
        if test_suites[issue_id].run(patch).passes:
            resolved += 1
    return (resolved / len(predictions)) * 100

Common pitfalls

• The 'plausible patch' distinction is explicitly discarded for repository-level evaluation; only patches that pass the full automated test suite count as correct.
• The localization metric is recall-oriented: the predicted file set must fully contain all developer-patched files, but may include additional false positives.
• Multimodal inputs like GIFs or videos require preprocessing via SSIM-based keyframe extraction before being fed to the LLM.

Evidence (verbatim from paper)

For evaluation on SWE-bench Lite and SWE-bench Multimodal, we adopt the established metrics from prior work: (1) %Resolved - the percentage of issues successfully fixed in the benchmark. Notably, for repository-level evaluation, all patches are validated via automated test suites; any patch that passes is considered correct, and the distinction of ”plausible patch” is no longer used. (2) Avg. $Cost - the average inference cost per tool run. Additionally, we introduce %Correct Localization - the percentage of cases where the tool’s localized files exactly match those modified in developer patches.

Citation

@misc{pan2026reporepair,
  title={RepoRepair: Leveraging Code Documentation for Repository-Level Automated Program Repair},
  author={Zhongqiang Pan et al. (2026)},
  year={2026},
  note={arXiv:2603.01048}
}

• arXiv: 2603.01048