Repobench R Eval

repobench-r-eval

RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems — Liu et al. (2023) (arXiv:2306.03091, 2023)

What this evaluates

Evaluates a model's ability to retrieve relevant cross-file code snippets given an in-file context for predicting the next line of code.

Datasets

• RepoBench-R — total ?; splits: test (-1)

Metrics

• acc@1 (primary) — range: [0, 1]
  • Accuracy at top-1 retrieval. 1 if the gold snippet is ranked first, 0 otherwise.
• acc@3 — range: [0, 1]
  • Accuracy at top-3 retrieval.
• acc@5 — range: [0, 1]
  • Accuracy at top-5 retrieval.

Input / output format

Input: Cropped in-file code (typically last 3 lines) and a pool of candidate code snippets from other files in the repository.

Output: Top-k ranked candidate code snippets.

Scoring recipe

def score(predictions, gold):
    acc1 = 1.0 if gold in predictions[:1] else 0.0
    acc3 = 1.0 if gold in predictions[:3] else 0.0
    acc5 = 1.0 if gold in predictions[:5] else 0.0
    return acc1, acc3, acc5

Common pitfalls

• Random retrieval baseline requires averaging over 100 runs for stability.
• Cropping m=3 lines is standard but ablation studies show sensitivity to the number of kept lines.

Evidence (verbatim from paper)

Table 2 presents a detailed comparison of different retrieval strategies in RepoBench-R. | acc@1 | acc@3 | acc@5 |

Citation

@misc{liu2023repobench,
  title={RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2306.03091}
}

• arXiv: 2306.03091