Xcodeeval Ruby Eval

xcodeeval-ruby-eval

Collaborative Agents for Automated Program Repair in Ruby — Akbarpour et al. (2025) (arXiv:2511.03925, 2025)

What this evaluates

Evaluates the ability of multi-agent LLM frameworks to automatically fix buggy Ruby code using test-driven feedback loops. It probes iterative code repair, self-reflection, and test generation capabilities under varying difficulty levels and error types.

Datasets

• xCodeEval (Ruby subset) — total ?; splits: validation (-1)

Metrics

• pass@1 (primary) — range: percent
  • The percentage of tasks where the generated code passes all hidden unit tests. The paper also reports cumulative pass@1, which sums the number of tasks solved at least once up to a given iteration across all prior iterations.

Input / output format

Input: Problem description, buggy Ruby code, input/output specifications, time/memory limits, and optionally sample input-output pairs.

Output: Fixed Ruby code.

Scoring recipe

solved_count = 0
for task in dataset:
    generated_code = model.generate(task)
    if generated_code.passes_all_hidden_tests(task.hidden_tests):
        solved_count += 1
pass_at_1 = (solved_count / len(dataset)) * 100

Common pitfalls

• Iterative repair can cause regressions where previously solved tasks become unsolved in later iterations.
• Cumulative pass@1 tracks tasks solved at least once across iterations, which differs from standard single-attempt pass@1.
• False positives in generated tests are more detrimental than false negatives because they halt further repair attempts.

Evidence (verbatim from paper)

Due to resource limitations, we used a 10% sampled subset of the dataset for all experiments in RQ1 (including RAMP and all the baselines) corresponding to 34 questions for Ruby. For all other research questions, which are evaluated only on RAMP, we use the full validation set. Additionally, for all experiments including RAMP and baselines (except the left plot of Figure[3]), we report cumulative pass@1, where the total number of tasks that have been solved at least once up to a given iteration is summed across all prior iterations and reported.

Citation

@misc{akbarpour2025ramp,
  title={Collaborative Agents for Automated Program Repair in Ruby},
  author={Akbarpour et al. (2025)},
  year={2025},
  note={arXiv:2511.03925}
}

• arXiv: 2511.03925