Tbar Eval

tbar-eval

TBar: Revisiting Template-based Automated Program Repair — Liu et al. (2019) (arXiv:1903.08409, 2019)

What this evaluates

Evaluates the effectiveness of template-based automated program repair systems by applying fix patterns to buggy Java programs. It probes the system's ability to localize faults, generate syntactically valid patches, and pass test suites without breaking existing tests.

Datasets

• Defects4J — total 395; splits: test (395)

Metrics

• plausible_patch (primary) — range: [0, 1]
  • A patch is considered plausible if the patched program passes all previously failing test cases without failing any previously passing test cases. The metric reports the fraction of bugs for which at least one plausible patch is generated.
• correct_patch — range: [0, 1]
  • A plausible patch is considered correct if it is manually verified to be semantically equivalent to the developer-provided ground-truth patch.

Input / output format

Input: Buggy Java program, associated test suite (categorized as passing or failing), and developer-provided ground-truth patch (used only for evaluation).

Output: A patched Java source file, or null if no plausible patch is generated within the time limit.

Scoring recipe

def evaluate_patch(prediction, test_suite, gold_patch):
    if prediction is None:
        return {"plausible": False, "correct": False}
    if passes_all_tests(prediction, test_suite):
        plausible = True
        correct = is_semantically_equivalent(prediction, gold_patch)
        return {"plausible": plausible, "correct": correct}
    return {"plausible": False, "correct": False}

Common pitfalls

• Confusing 'plausible' (passes tests) with 'correct' (semantically equivalent to developer fix).
• TBar stops generating patches after finding the first plausible patch per bug, unlike systems that enumerate all plausible patches.
• Evaluation is subject to a strict 3-hour timeout per bug, which caps the maximum achievable fix rate.

Evidence (verbatim from paper)

If the patched program passes all tests successfully, the patch candidate is considered as a plausible patch. Once such a plausible patch is identified, TBar stops generating other patch candidates for this bug to fix bugs in a standard and practical program repair workflow... If a plausible patch is generated, we further manually check the equivalence between this patch and the ground-truth patch provided by developers and available in the Defects4J benchmark. If the plausible patch is semantically equivalent to the ground-truth patch, the plausible patch is considered as correct.

Citation

@misc{liu2019tbar,
  title={TBar: Revisiting Template-based Automated Program Repair},
  author={Liu et al. (2019)},
  year={2019},
  note={arXiv:1903.08409}
}

• arXiv: 1903.08409