Execrepo Bench Eval

execrepo-bench-eval

ExecRepoBench: Multi-level Executable Code Completion Evaluation — Yang et al. (2024) (arXiv:2412.11990, 2024)

What this evaluates

Evaluates repository-level code completion capabilities of LLMs across multiple granularities (span, line, expression, statement, function) using executable validation and string similarity metrics.

Datasets

• ExecRepoBench — total 1200; splits: test (1200)

Metrics

• Pass@1 (primary) — range: percent
  • Percentage of generated code completions that pass all associated unit tests when evaluated with greedy decoding.
• Edit Similarity (ES) — range: percent
  • String-based similarity score calculated by comparing the generated code against the ground-truth code using edit distance.

Input / output format

Input: Repository-level code snippets with masked sections at various granularities (span, single-line, multi-line, expression, statement, function) requiring the model to complete the code.

Output: Completed code snippet to replace the masked section.

Scoring recipe

# For each sample in the benchmark:
generated_code = model.predict(context)
ground_truth_code = sample.gold
unit_tests = sample.tests

# Edit Similarity
es_score = 1 - edit_distance(generated_code, ground_truth_code) / max(len(generated_code), len(ground_truth_code))

# Pass@1 (Greedy)
test_results = run_unit_tests(generated_code, unit_tests)
pass_at_1 = 1.0 if all(test_results) else 0.0

# Aggregate
avg_es = mean(es_scores)
avg_pass_at_1 = mean(pass_at_1_scores)

Common pitfalls

• High ES scores do not guarantee functional correctness, as semantic equivalence may exist without exact string matching.
• Greedy decoding for Pass@1 may underestimate model capability compared to sampling-based Pass@k evaluations.
• Repository-level completion requires handling cross-file dependencies, which standard in-file benchmarks ignore.

Evidence (verbatim from paper)

We compare the generated code and the ground-truth code using edit similarity (ES) to report string-based scores. Similar to the in-file benchmark HumanEval/MBPP, we employ the Pass@k metric based on the executable results to get the reliability evaluation results. In this work, we report the greedy Pass@1 score of all LLMs with greedy inference for a fair comparison.

Citation

@misc{yang2024execrepo,
  title={ExecRepoBench: Multi-level Executable Code Completion Evaluation},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2412.11990}
}

• arXiv: 2412.11990