Swebench Eval

swebench-eval

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Cobbe et al. (2023) (arXiv:2310.06770, 2023)

What this evaluates

Evaluates language models' ability to resolve real-world software engineering issues by generating code patches. It probes long-context reasoning, cross-file dependency understanding, and execution-based validation within large, complex codebases.

Datasets

• SWE-bench — total ?; splits: (unstated)

Metrics

• resolve_rate (primary) — range: percent
  • Percentage of benchmark instances where the model's generated patch successfully resolves the issue, typically validated by passing all associated repository tests.

Input / output format

Input: Task instructions, issue description text, retrieved source files (via BM25 or oracle retrieval), documentation, an example patch file, and a prompt instructing the model to generate a patch.

Output: A code patch file intended to resolve the specified GitHub issue.

Scoring recipe

resolved = 0
for instance in dataset:
    patch = model.generate(instance.input)
    if apply_and_run_tests(patch, instance.repo) == 'PASS_ALL':
        resolved += 1
resolve_rate = (resolved / len(dataset)) * 100

Common pitfalls

• Context window limits force aggressive file retrieval, which may exclude necessary context or include irrelevant files.
• Oracle retrieval assumes knowledge of edited files, making it unrealistic for actual engineering workflows.
• Tokenization differences across models (e.g., Llama vs GPT-4) cause non-standard context length measurements, affecting fair comparison.

Evidence (verbatim from paper)

SWE-bench instances provide an issue description and a codebase as input to the model... we construct the input to the model consisting of task instructions, the issue text, retrieved files and documentation, and finally an example patch file and prompt for generating the patch file. Table 2: Model resolve rates with BM25 retrieval, with different maximum context lengths.

Citation

@misc{cobbe2023swebench,
  title={SWE-bench: Can Language Models Resolve Real-World GitHub Issues?},
  author={Cobbe et al. (2023)},
  year={2023},
  note={arXiv:2310.06770}
}

• arXiv: 2310.06770