Csr Bench Eval

csr-bench-eval

CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories — Xiao et al. (2025) (arXiv:2502.06111, 2025)

What this evaluates

Evaluates LLM agents' ability to autonomously deploy computer science research repositories by executing multi-stage workflows including setup, downloading dependencies, training models, running evaluations, and performing inference. It probes instruction comprehension, command generation, and iterative error correction in complex software environments.

Datasets

• CSR-Bench — total 100; splits: test (100)

Metrics

• success rate (primary) — range: [0, 1]
  • The proportion of repositories that successfully complete a given deployment stage (Setup, Download, Training, Evaluation, or Inference) out of the total number of repositories tested for that stage.

Input / output format

Input: Natural language instructions for deploying a specific research repository, along with access to execution tools (shell commands, log analysis, issue retrieval, web search) and the repository's source code.

Output: A sequence of executed commands and script modifications that successfully complete the target deployment stage.

Scoring recipe

def compute_success_rate(stage_results, total_instances):
    successful = sum(1 for res in stage_results if res.success)
    return successful / total_instances

Common pitfalls

• Models struggle significantly with complex stages like Training, Evaluation, and Inference compared to basic Setup and Download, often failing due to environment configuration nuances.
• Hardware and software compatibility issues are common and frequently cause failures even when models have access to advanced tools like web search or issue trackers.

Evidence (verbatim from paper)

As shown in Table[1], all models perform well on Setup and Download tasks (success rates around 0.23 to 0.28) but struggle with Training, Evaluation, and Inference, where success rates are close to zero.

Citation

@misc{xiao2025csrbench,
  title={CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories},
  author={Xiao et al. (2025)},
  year={2025},
  note={arXiv:2502.06111}
}

• arXiv: 2502.06111