Repobench C Eval

repobench-c-eval

RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems — Liu et al. (2023) (arXiv:2306.03091, 2023)

What this evaluates

Evaluates autoregressive language models on predicting the next line of code using provided in-file and cross-file contexts.

Datasets

• RepoBench-C — total ?; splits: test (-1)

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • 1 if the predicted line exactly matches the gold line, 0 otherwise.
• Edit Similarity (ES) — range: [0, 1]
  • Token-level edit similarity between predicted and gold lines.

Input / output format

Input: In-file context (import statements and preceding lines) and cross-file context (snippets from other files parsed by import statements).

Output: Predicted next line of code.

Scoring recipe

def score(pred, gold):
    em = 1.0 if pred.strip() == gold.strip() else 0.0
    es = edit_similarity(pred, gold)  # token-level
    return em, es

Common pitfalls

• Model-language pairings must be strictly matched (e.g., CodeGen-Mono for Python, CodeGen-Multi for Java).
• Context length limits (2k vs 8k) significantly impact performance and can cause out-of-distribution issues.

Evidence (verbatim from paper)

Table 3: RepoBench-C performance ... evaluated using Exact Match (EM), Edit Similarity (ES).

Citation

@misc{liu2023repobench,
  title={RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2306.03091}
}

• arXiv: 2306.03091