Tam Eval

tam-eval

TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance — Bruches et al. (2026) (arXiv:2601.18241, 2026)

What this evaluates

Evaluates LLMs on automated unit test maintenance tasks, including test creation, repair, and updating across Python, Java, and Go. Probes context-aware reasoning, code generation, and the ability to dynamically adapt tests to production code changes.

Datasets

• TAM-Eval — total 1539; splits: test (1539); repo https://github.com/trndcenter/TAM-Eval

Metrics

• pass_rate — range: percent
  • Percentage of test cases that execute successfully without compilation or runtime errors.
• line_coverage — range: percent
  • Percentage of executable lines in the focal function covered by the generated test suite.
• mutation_score (primary) — range: percent
  • Percentage of syntactically or semantically mutated code variants (mutants) killed by the generated test suite.

Input / output format

Input: Focal production file (and optionally a baseline test file for repair/update tasks), along with task-specific instructions (creation, repair, or updating).

Output: A complete test file (code) containing test cases for the focal function.

Scoring recipe

def evaluate(prediction_code, focal_code, task_type):
    if not compile(prediction_code): return {"pass_rate": 0, "line_coverage": 0, "mutation_score": 0}
    pass_rate = run_tests_and_check_stability(prediction_code)
    line_coverage = measure_line_coverage(prediction_code, focal_code)
    mutation_score = kill_mutants(prediction_code, focal_code)
    return {"pass_rate": pass_rate, "line_coverage": line_coverage, "mutation_score": mutation_score}

Common pitfalls

• Models may generate syntactically valid tests that pass but fail to improve coverage or mutation score (superficial coverage).
• Flaky or non-deterministic tests can artificially inflate pass rates; the protocol requires consistent execution across two runs.
• High comment density in focal files can cause data leakage, leading models to generate tests based on documentation rather than executable logic.

Evidence (verbatim from paper)

Test coverage level. We exclude focal-test file pairs where the original test suite achieves less than 40% line coverage of the corresponding focal function. This ensures that retained test suites are not trivially under-specified and that there is a meaningful baseline from which to assess improvements in coverage and mutation score.

Citation

@misc{bruches2026tam_eval,
  title={TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance},
  author={Bruches et al. (2026)},
  year={2026},
  note={arXiv:2601.18241}
}

• arXiv: 2601.18241