LLM Apr Eval

llm-apr-eval

Empirical Evaluation of Generalizable Automated Program Repair with Large Language Models — Campos et al. (2025) (arXiv:2506.03283, 2025)

What this evaluates

Evaluates the ability of large language models to automatically generate correct code patches for buggy functions across Java, JavaScript, Python, and PHP. It probes language-specific repair capabilities, the impact of providing test case information, and the sensitivity to fault localization granularity.

Datasets

• Defects4J — total ?; splits: test (-1)
• BugsInPy — total ?; splits: test (-1)
• BugsJS — total ?; splits: test (-1)
• BugsPHP — total ?; splits: test (-1)

Metrics

• plausible@1 (primary) — range: percent
  • Success rate when generating a single plausible patch. A patch is considered plausible if it passes all provided test cases without introducing new failures. Computed as the fraction of problems where the top-1 generated patch is plausible.
• plausible@5 — range: percent
  • Success rate among the top 5 generated candidates. Computed as the fraction of problems where at least one of the top-5 generated patches is plausible.
• p_combined@k — range: percent
  • Probability that at least one model in a committee produces a plausible top-k result. Formula: E[1 - prod_{i=1 to m} (C(n-c_i, k) / C(n, k))], where n is total problems, c_i is correct generations by model i, and m is committee size.

Input / output format

Input: Buggy function code (base prompt), optionally augmented with line-level fault localization hints (comments marking suspicious lines) and/or names, source code, and error messages of failing test cases.

Output: A single code patch (function replacement or diff) intended to fix the buggy function.

Scoring recipe

def compute_plausible_k(predictions, test_cases, k=1):
    plausible_count = 0
    for problem_preds in predictions:
        top_k_patches = problem_preds[:k]
        if any(patch_passes_tests(patch, test_cases) for patch in top_k_patches):
            plausible_count += 1
    return (plausible_count / len(predictions)) * 100

Common pitfalls

• Python indentation errors in generated patches frequently cause execution failures, artificially lowering plausible scores unless a post-processing linting/formatting step is applied.
• Assuming perfect fault localization (method- or line-level) overestimates real-world performance, as noisy tool-generated localization causes significant accuracy drops.
• Model performance is highly language-specific; averaging across languages without stratification masks specialization and committee complementarity effects.

Evidence (verbatim from paper)

Table II summarizes the performance of all models across the four benchmarks, reporting plausible@1 (the success rate when generating a single fix) and plausible@5 (success among five generated candidates).

Citation

@misc{campos2025empirical,
  title={Empirical Evaluation of Generalizable Automated Program Repair with Large Language Models},
  author={Campos et al. (2025)},
  year={2025},
  note={arXiv:2506.03283}
}

• arXiv: 2506.03283