Multi Swe Bench Eval

multi-swe-bench-eval

Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving — Zan et al. (2025) (arXiv:2504.02605, 2025)

What this evaluates

This benchmark evaluates an LLM's ability to resolve software engineering issues across multiple programming languages. It probes capabilities in long-context reasoning, multi-file code patching, and fault localization by requiring models to generate executable fixes for real-world repository issues.

Datasets

• Multi-SWE-bench — total 1632; splits: test (1632); repo https://github.com/multi-swe-bench/multi-swe-bench

Metrics

• Resolved Rate (%) (primary) — range: percent
  • Percentage of issues for which the model's generated patch successfully resolves the reported bug or feature request, typically validated via reproduction tests.
• Success Location (%) — range: percent
  • Accuracy of fault localization at the file level, measuring whether the model correctly identifies the file containing the bug.
• Average Cost ($) — range: other
  • Average monetary cost incurred per issue resolved, accounting for API calls and execution resources.

Input / output format

Input: Per instance, the model receives the issue description, full file contents of the target repository, and a pruned repository directory structure. Language-specific environment setup and test commands are also provided.

Output: The model must produce a code patch (diff) or a sequence of file edits intended to resolve the reported issue, which can be applied via git or direct file modification.

Scoring recipe

def evaluate(predictions, gold_instances):
    resolved = 0
    loc_correct = 0
    total_cost = 0.0
    for pred, gold in zip(predictions, gold_instances):
        if apply_patch_and_run_tests(pred.patch, gold.repo, gold.test_script):
            resolved += 1
        if pred.located_file == gold.target_file:
            loc_correct += 1
        total_cost += pred.api_cost
    return {
        'Resolved Rate (%)': (resolved / len(gold_instances)) * 100,
        'Success Location (%)': (loc_correct / len(gold_instances)) * 100,
        'Average Cost ($)': total_cost / len(gold_instances)
    }

Common pitfalls

• Extracting file skeletons is challenging in some programming languages, requiring full file content inputs that may exceed LLM context limits.
• Compiled artifacts (e.g., .o, .bin) can interfere with git apply if not explicitly excluded via .gitignore.
• Regression and reproduction testing is cumbersome to implement across languages, so patch validation pipelines must be carefully adapted per language.

Evidence (verbatim from paper)

Following SWE-Bench*[Jimenez et al., [2023]]* and SWE-Lancer*[Miserendino et al., [2025]]*, we adopt Resolved Rate (%) as our primary evaluation metric, measuring the percentage of issues resolved. In addition, we report several other metrics to provide a more detailed analysis: Success Location (%) — the accuracy of fault localization at file level; and Average Cost ($) — the average cost per issue.

Citation

@misc{zan2025multi,
  title={Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving},
  author={Zan et al. (2025)},
  year={2025},
  note={arXiv:2504.02605}
}

• arXiv: 2504.02605