Codeflaws Repair Eval

codeflaws-repair-eval

Automated Repair of C Programs Using Large Language Models — Farzandway et al. (2025) (arXiv:2509.01947, 2025)

What this evaluates

This evaluation probes an LLM's ability to automatically detect and fix bugs in C programs by generating correct patches. It measures how effectively the model leverages test feedback, fault localization scores, and iterative reasoning to pass all provided test cases for each buggy submission.

Datasets

• Codeflaws — total 3902; splits: test (3902)

Metrics

• Repair Accuracy (primary) — range: percent
  • A bug is considered correctly repaired if the generated patch passes all test cases. Calculated as the fraction of bugs where the patch passes every provided test.
• Time-to-Repair — range: other
  • Total elapsed time including fault localization, patch generation, and validation.
• Partial Repair Rate — range: percent
  • The proportion of errors where the repair attempt produces a patch that passes some, but not all, of the previously failing test cases.

Input / output format

Input: Buggy C source code, associated test cases with pass/fail results, expected outputs, and optionally SBFL suspiciousness scores, compiler/runtime logs, and historical debugging context.

Output: Corrected C source code patch.

Scoring recipe

def compute_metrics(predictions, gold_tests):
    total = len(predictions)
    repaired = 0
    partially_repaired = 0
    for patch, tests in zip(predictions, gold_tests):
        results = run_tests(patch, tests)
        if all(results):
            repaired += 1
        elif any(results):
            partially_repaired += 1
    accuracy = repaired / total
    partial_rate = partially_repaired / total
    return accuracy, partial_rate

Common pitfalls

• Evaluating on only a subset of the provided test cases instead of requiring the patch to pass all of them for a positive repair accuracy score.
• Ignoring compilation failures or infinite loops, which should be treated as test failures or handled with strict timeouts (e.g., 2 minutes per test).
• Counting a bug as repaired if it passes only the initially failing tests, rather than verifying it passes the entire test suite including previously passing cases.

Evidence (verbatim from paper)

We evaluated the system’s performance using the following metrics: Repair Accuracy: A bug is considered correctly repaired if the generated patch passes all test cases. Time-to-Repair: Measured as the total elapsed time including fault localization, patch generation, and validation. Partial Repair Rate: The proportion of errors where the repair attempt produces a patch that passes some, but not all, of the previously failing test cases.

Citation

@misc{farzandway2025automated,
  title={Automated Repair of C Programs Using Large Language Models},
  author={Farzandway et al. (2025)},
  year={2025},
  note={arXiv:2509.01947}
}

• arXiv: 2509.01947