# Repobench C Eval

> Evaluates autoregressive language models on predicting the next line of code using provided in-file and cross-file contexts. Use when the user wants to benchmark on RepoBench-C, or asks about evaluating this task. Reports Exact Match (EM).

- Skill: `qhjqhj00/repobench-c-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/repobench-c-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/repobench-c-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/repobench-c-eval

---


# repobench-c-eval

> RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems — Liu et al. (2023) (arXiv:2306.03091, 2023)

## What this evaluates

Evaluates autoregressive language models on predicting the next line of code using provided in-file and cross-file contexts.

## Datasets

- **RepoBench-C** — total ?; splits: test (-1)

## Metrics

- `Exact Match (EM)` **(primary)** — range: [0, 1]
  - 1 if the predicted line exactly matches the gold line, 0 otherwise.
- `Edit Similarity (ES)` — range: [0, 1]
  - Token-level edit similarity between predicted and gold lines.

## Input / output format

**Input**: In-file context (import statements and preceding lines) and cross-file context (snippets from other files parsed by import statements).

**Output**: Predicted next line of code.

## Scoring recipe

```python
def score(pred, gold):
    em = 1.0 if pred.strip() == gold.strip() else 0.0
    es = edit_similarity(pred, gold)  # token-level
    return em, es
```

## Common pitfalls

- Model-language pairings must be strictly matched (e.g., CodeGen-Mono for Python, CodeGen-Multi for Java).
- Context length limits (2k vs 8k) significantly impact performance and can cause out-of-distribution issues.

## Evidence (verbatim from paper)

> Table 3: RepoBench-C performance ... evaluated using Exact Match (EM), Edit Similarity (ES).

## Citation

```bibtex
@misc{liu2023repobench,
  title={RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2306.03091}
}
```

- arXiv: 2306.03091