Repobench P Eval

repobench-p-eval

RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems — Liu et al. (2023) (arXiv:2306.03091, 2023)

What this evaluates

Evaluates an end-to-end pipeline that first retrieves cross-file snippets and then predicts the next line of code using both the in-file context and retrieved snippets.

Datasets

• RepoBench-P — total ?; splits: test (-1)

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • 1 if the predicted line exactly matches the gold line, 0 otherwise.
• Edit Similarity (ES) — range: [0, 1]
  • Token-level edit similarity between predicted and gold lines.

Input / output format

Input: In-file context (up to 1600 tokens, max 60 lines) and retrieved cross-file snippets (up to 6400 total tokens).

Output: Predicted next line of code.

Scoring recipe

def score(pred, gold):
    em = 1.0 if pred.strip() == gold.strip() else 0.0
    es = edit_similarity(pred, gold)
    return em, es

Common pitfalls

• Retrieval strategies (Gold-Only, Gold-Filled, UniXcoder, Random, Baseline) drastically affect results.
• Gold snippets are included for reference only and do not represent oracle capabilities.

Evidence (verbatim from paper)

Table 4: Comparison of various retrieval strategies on the RepoBench-P ... evaluated in terms of Exact Match (EM) and Edit Similarity (ES) metrics

Citation

@misc{liu2023repobench,
  title={RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2306.03091}
}

• arXiv: 2306.03091