Repocoder Eval

repocoder-eval

RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation — Fengji Zhang et al. (2023) (arXiv:2303.12570, 2023)

What this evaluates

This benchmark evaluates repository-level code completion by measuring how accurately a model predicts missing code segments given surrounding context and retrieved repository snippets. It probes both syntactic similarity and functional correctness across line, API, and function-level granularity.

Datasets

• RepoEval — total ?; splits: test (-1)

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • Binary metric; returns 1 if the predicted code exactly matches the ground truth, and 0 otherwise.
• Edit Similarity (ES) — range: [0, 1]
  • Fine-grained similarity calculated as 1 - (Levenshtein distance between prediction and ground truth) / max(len(prediction), len(ground truth)).
• Pass Rate (PR) — range: [0, 1]
  • Functional correctness metric; returns 1 if the generated code passes all corresponding unit tests, and 0 otherwise.

Input / output format

Input: Unfinished code context (prompt) and optionally retrieved code snippets from the repository.

Output: Predicted code completion text string (Ŷ).

Scoring recipe

def compute_metrics(predictions, golds, unit_tests=None):
    em_scores = [1.0 if p == g else 0.0 for p, g in zip(predictions, golds)]
    es_scores = []
    for p, g in zip(predictions, golds):
        lev = levenshtein_distance(p, g)
        es_scores.append(1.0 - lev / max(len(p), len(g)))
    if unit_tests:
        pr_scores = [1.0 if run_tests(p, unit_tests) else 0.0 for p in predictions]
    else:
        pr_scores = None
    return {'EM': sum(em_scores)/len(em_scores), 'ES': sum(es_scores)/len(es_scores), 'PR': pr_scores}

Common pitfalls

• Relying solely on EM/ES for function completion ignores functional correctness; unit test execution is required for accurate assessment.
• Hyperparameters like sliding window size (Sw) and retrieval count (K) are highly task-dependent and require adjustment for different programming languages or contexts.
• The Oracle baseline uses ground-truth code for retrieval, which provides an unrealistic upper bound that cannot be replicated in real-world deployment.

Evidence (verbatim from paper)

Similarity-based Evaluation: Following established practices in code completion research (Lu et al., 2021, 2022), we evaluate our line and API completion datasets using two metrics: Exact Match (EM) and Edit Similarity (ES). The EM score is a binary metric that takes the value of 1 if the predicted code exactly matches the ground truth code, and 0 otherwise. The ES metric provides a more fine-grained evaluation and is calculated as $ES = 1 - \frac{\operatorname{Lev}(\hat{Y}, Y)}{\max(|\hat{Y}|, |Y|)}$, where Lev represents the Levenshtein distance (Levenshtein et al., 1966).

Citation

@misc{zhang2023repocoder,
  title={RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation},
  author={Fengji Zhang et al. (2023)},
  year={2023},
  note={arXiv:2303.12570}
}

• arXiv: 2303.12570