Swe Bench Eval

swe-bench-eval

SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering — Yang et al. (2024) (arXiv:2405.15793, 2024)

What this evaluates

Evaluates an agent's ability to automatically resolve software engineering issues by generating and applying code patches to open-source repositories. It measures functional correctness by running the target repository's test suite after the patch is applied.

Datasets

• SWE-bench — total 2294; splits: test (2294), lite (300)
• HumanEvalFix — total ?; splits: test (-1)

Metrics

• pass@1 (primary) — range: percent
  • The proportion of task instances for which all repository tests pass successfully after the model's generated patch is applied to the codebase.
• Avg. Cost — range: other
  • The average API inference cost incurred by the agent, calculated only over instances that were successfully resolved.

Input / output format

Input: System prompt, issue description, optionally a demonstration, and the full codebase context.

Output: A generated patch file or series of code edits applied to the repository.

Scoring recipe

resolved_count = 0
for pred in predictions:
    apply_patch_to_repo(pred)
    if run_repo_tests().all_pass:
        resolved_count += 1
return (resolved_count / len(predictions)) * 100

Common pitfalls

• The evaluation requires running the full test suite of the target repository, which can be computationally expensive and sensitive to environment setup.
• A per-instance budget cap ($4) is enforced; runs exceeding this budget are automatically submitted, which may artificially lower the pass rate if the agent hasn't finished editing.

Evidence (verbatim from paper)

We report % Resolved or pass@1 as the main metric, which is the proportion of instances for which all tests pass successfully after the model generated patch is applied to the repository [20].

Citation

@misc{yang2024sweagent,
  title={SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2405.15793}
}

• arXiv: 2405.15793