Swe Rebench V2 Eval

swe-rebench-v2-eval

SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale — Badertdinov et al. (2026) (arXiv:2602.23866, 2026)

What this evaluates

Evaluates the ability of LLM-based agents to autonomously resolve software engineering issues by modifying code in real-world repositories. It probes environment setup, code generation, and test execution capabilities across multiple programming languages.

Datasets

• SWE-rebench V2 — total 32000; splits: test (300)

Metrics

• pass@1 (primary) — range: percent
  • Fraction of tasks where at least one out of k attempts successfully passes the test suite.

Input / output format

Input: Repository codebase, issue description, and test suite provided in an interactive shell environment with base dependencies installed.

Output: Modified source code files and/or shell commands to resolve the issue. Success is determined by running the provided test suite.

Scoring recipe

def compute_pass_at_k(successes_per_task, k):
    # successes_per_task: list of lists of booleans (k attempts per task)
    passed = [any(attempts) for attempts in successes_per_task]
    return sum(passed) / len(passed)

Common pitfalls

• Test suite coupling: models may correctly fix the target issue but fail due to regressions in unrelated code paths caught by the test suite.
• Implicit naming requirements: tests often expect specific implementation details not specified in the problem statement.
• External dependencies: problem statements may reference inaccessible URLs or auth-walled resources, causing agent failure.

Evidence (verbatim from paper)

Table 6: Pass rates (%) by model and programming language. | Model | Py | JS | Go | Rust | Scala | pass@1 | pass@3 |

Citation

@misc{badertdinov2026swe_rebench_v2,
  title={SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale},
  author={Badertdinov et al. (2026)},
  year={2026},
  note={arXiv:2602.23866}
}

• arXiv: 2602.23866