Swe Bench Eval

swe-bench-eval — Evaluate code agents on real GitHub issues

Source: Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (arXiv:2310.06770).

What this benchmark tests

Given a real GitHub issue + the state of the codebase before the fix PR, the model must produce a patch (unified diff) that, when applied, makes the original PR's hidden test suite pass. This exercises long-context reasoning, cross-file edits, and execution-grounded correctness — far beyond function-level codegen.

When to invoke this skill

• User says "evaluate / benchmark / test on SWE-Bench"
• User asks how good a coding agent is at "real bug fixes" / "GitHub issues" / "multi-file edits"
• User mentions any variant: SWE-Bench, SWE-Bench Lite, SWE-Bench Verified, SWE-Bench Multimodal, SWE-Bench Live

Variants — pick before running

Default to _Verified unless the user wants the full sweep.

Setup

pip install swebench  # official harness with docker eval images

Each instance has: repo, instance_id, base_commit, problem_statement (issue text), hints_text, test_patch (the gold tests, hidden from the model), patch (the gold diff, hidden from the model), FAIL_TO_PASS / PASS_TO_PASS test lists.

Evaluation protocol (verbatim from §4 + §5)

1. Input to the agent: problem_statement + a way to read the codebase at base_commit. Either:
   • oracle retrieval — give the model exactly the files edited in gold patch (upper-bound study only)
   • bm25 retrieval — top-k files by BM25 over issue text (paper's main setting)
   • agentic — let the agent navigate the repo itself (most modern submissions)
2. Output: a model_patch field (unified diff against base_commit).
3. Scoring: swebench harness builds a docker image per instance, applies model_patch, then runs FAIL_TO_PASS + PASS_TO_PASS tests. Instance is resolved iff all FAIL_TO_PASS pass AND all PASS_TO_PASS still pass.
4. Headline metric: resolve_rate = resolved / total. Always report alongside applied_rate (% patches that even apply cleanly) for diagnosis.

Reference scoring (sketch)

# 1. Save predictions as JSONL: one line per instance with model_name_or_path + instance_id + model_patch
# 2. Run the harness:
python -m swebench.harness.run_evaluation \
    --dataset_name princeton-nlp/SWE-bench_Verified \
    --predictions_path predictions.jsonl \
    --max_workers 8 \
    --run_id my-eval-2026
# Outputs: ./logs/run_evaluation/my-eval-2026/<model>/results.json with resolve_rate.

Output format the agent must produce

{"instance_id": "django__django-12345", "model_name_or_path": "claude-opus-4-7", "model_patch": "diff --git a/...\n--- a/...\n+++ b/...\n@@ ..."}

Failure modes that produce 0% resolve rate even with smart edits:

• Patch references file paths that don't exist at base_commit (model hallucinated structure)
• Patch breaks PASS_TO_PASS tests (regression)
• Patch doesn't apply because of context drift (whitespace / line numbers off)

Don'ts

• Don't compute resolve_rate yourself by diffing predictions against the gold patch — patches that look different can still resolve the issue. Always run the harness.
• Don't conflate _Lite / _Verified / full numbers in the same table without labelling.
• Don't forget PASS_TO_PASS; without that you're rewarding regressions.

Citation

@inproceedings{jimenez2024swebench,
  title={SWE-bench: Can Language Models Resolve Real-World GitHub Issues?},
  author={Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik},
  booktitle={ICLR}, year={2024}
}