Swe Bench Java Eval

swe-bench-java-eval

SWE-bench-java: A GitHub Issue Resolving Benchmark for Java — Zan et al. (2024) (arXiv:2408.14354, 2024)

What this evaluates

This benchmark evaluates an AI agent's ability to autonomously resolve real-world GitHub issues in Java projects. It probes capabilities in code patch generation, repository navigation, test case reasoning, and handling runtime environment dependencies.

Datasets

• SWE-bench-java-verified — total 137; splits: test (137)

Metrics

• Resolved Rate (%) (primary) — range: percent
  • The proportion of issues successfully resolved out of the total dataset. An issue is marked resolved only if all provided test cases pass after applying the model's generated patch.

Input / output format

Input: GitHub issue description, full repository codebase, and a set of failing test cases associated with the issue.

Output: A code patch (modified source files) intended to fix the issue and make all associated test cases pass.

Scoring recipe

resolved_count = 0
for instance in dataset:
    patch = model.generate_patch(instance.issue, instance.repo, instance.tests)
    if run_tests(instance.repo, patch, instance.tests).all_pass:
        resolved_count += 1
return (resolved_count / len(dataset)) * 100

Common pitfalls

• Runtime environment must be correctly configured for each Java issue; missing setup prevents accurate issue reproduction and artificially lowers scores.
• Resolution requires ALL given test cases to pass, not just a subset or the main failing test.
• The benchmark evaluates agentic workflows (SWE-agent) rather than direct code completion, so evaluation depends on the agent's ability to navigate, edit, and execute tests autonomously.

Evidence (verbatim from paper)

Following SWE-bench [1], we adopt the Resolved Rate (%) as our evaluation metric. This metric indicates the proportion of issues in SWE-bench-java-verified that are successfully resolved. An issue is considered resolved only if all given test cases pass. This metric provides a precise measure of effectiveness in resolving real-world Java GitHub issues.

Citation

@misc{zan2024swebenchjava,
  title={SWE-bench-java: A GitHub Issue Resolving Benchmark for Java},
  author={Zan et al. (2024)},
  year={2024},
  note={arXiv:2408.14354}
}

• arXiv: 2408.14354