Swebench Live Eval

swebench-live-eval

SWE-bench Goes Live! — Zhang et al. (2025) (arXiv:2505.23419, 2025)

What this evaluates

Evaluates the ability of AI coding agents to autonomously resolve real-world software engineering issues by generating and applying patches to GitHub repositories. It probes cross-file reasoning, dependency management, and robustness against contamination from static benchmarks.

Datasets

• SWE-bench-Live — total 1319; splits: Lite (-1), Full (-1)

Metrics

• Resolved Rate (%) (primary) — range: percent
  • The proportion of issues successfully resolved by the agent across all task instances. Calculated as (number of resolved issues / total issues) * 100.
• Patch Apply Rate (%) — range: percent
  • The percentage of generated patches that are syntactically correct and can be successfully applied to the codebase without errors. Calculated as (number of successfully applied patches / total generated patches) * 100.
• Localization Success Rate (%) — range: percent
  • The percentage of patches where the set of files modified by the generated patch exactly matches the set of files in the gold patch. Calculated as (number of correctly localized patches / total patches) * 100.

Input / output format

Input: A target code repository paired with a GitHub issue description.

Output: A code patch (diff) intended to resolve the issue, specifying file modifications.

Scoring recipe

def compute_metrics(predictions, golds, total_instances):
    resolved = 0
    applied = 0
    localized = 0
    for pred, gold in zip(predictions, golds):
        if pred.is_resolved(gold): resolved += 1
        if pred.apply_successfully(): applied += 1
        if set(pred.modified_files) == set(gold.modified_files): localized += 1
    return {
        'Resolved Rate (%)': (resolved / total_instances) * 100,
        'Patch Apply Rate (%)': (applied / total_instances) * 100,
        'Localization Success Rate (%)': (localized / total_instances) * 100
    }

Common pitfalls

• Overfitting to static benchmarks like SWE-bench leads to inflated scores that do not generalize to live, unseen issues.
• Omitting infrastructure adaptation for regression testing or reranking stages can artificially inflate patch application and resolution rates.
• Difficulty scales non-linearly with patch scope and repository size, making small or simplified benchmark subsets misleading for real-world evaluation.

Evidence (verbatim from paper)

Following the evaluation protocol of SWE-bench, we adopt the Resolved Rate (%) as our primary metric. This measures the proportion of issues successfully resolved by the agent across all task instances. We also report the Patch Apply Rate (%), which indicates the percentage of generated patches that are syntactically correct and can be successfully applied to the codebase without errors. Additionally, we measure the Localization Success Rate (%) at the file level. This reflects whether the set of files modified by the generated patch matches the gold patch.

Citation

@misc{zhang2025swebenchgoeslive,
  title={SWE-bench Goes Live!},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2505.23419}
}

• arXiv: 2505.23419