Apr Plausible Patch Eval

apr-plausible-patch-eval

Impact of Code Language Models on Automated Program Repair — Jiang et al. (2023) (arXiv:2302.05020, 2023)

What this evaluates

Evaluates the ability of code language models to automatically generate correct patches for real-world Java bugs. It probes whether pre-trained or fine-tuned models can produce syntactically valid and semantically correct code that passes developer-written test suites and survives manual verification.

Datasets

• Defects4J v1.2 — total 130; splits: test (130)
• Defects4J v2.0 — total 108; splits: test (108)
• QuixBugs — total 40; splits: test (40)
• HumanEval-Java — total 164; splits: test (164)

Metrics

• plausible_patch (primary) — range: percent
  • Percentage of bugs where at least one of the 10 generated candidate patches passes all developer-written test cases and is manually verified as semantically correct.

Input / output format

Input: Buggy Java function with buggy lines either masked by a placeholder (e.g., ) or provided as comments, plus surrounding context.

Output: Patched code (either the whole patched function or just the patched lines plus the remainder of the function).

Scoring recipe

plausible_count = 0
for bug in bugs:
    patches = generate_patches(model, bug, k=10)
    for patch in patches:
        if run_test_cases(patch) == PASS:
            if manual_verify(patch):
                plausible_count += 1
                break
return plausible_count / len(bugs)

Common pitfalls

• Data leakage: Pre-trained CLMs may have encountered Defects4J/QuixBugs during pre-training, artificially inflating performance.
• Prompt sensitivity: Model performance varies significantly depending on whether buggy lines are masked or provided as comments.
• Test suite limitations: Passing developer-written tests only indicates syntactic/compilability correctness, not semantic correctness, requiring manual verification.

Evidence (verbatim from paper)

For all experiments, we let each tool (CLMs, fine-tuned CLMs, or DL-based APR techniques) generate ten candidate patches for each bug and run the developer-written test cases on the patched program. The first patched program that passes all the test cases is considered a plausible patch. And we finally manually check the correctness of plausible patches

Citation

@misc{jiang2023impact,
  title={Impact of Code Language Models on Automated Program Repair},
  author={Jiang et al. (2023)},
  year={2023},
  note={arXiv:2302.05020}
}

• arXiv: 2302.05020