R1 Code Interpreter Eval

r1-code-interpreter-eval

R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning — Chen et al. (2025) (arXiv:2505.21668, 2025)

What this evaluates

Probes LLMs' ability to autonomously generate and execute code for complex reasoning and planning tasks across logic, spatial, order, optimization, search, and math domains. It evaluates how well models can iteratively explore, optimize, and self-check solutions in multi-turn code execution environments.

Datasets

• SymBench — total ?; splits: test (-1)
• Big-Bench-Hard — total ?; splits: test (-1)
• Reasoning-Gym — total ?; splits: test (-1); repo https://github.com/open-thought/reasoning-gym

Metrics

• exact match or constraint check (primary) — range: [0, 1]
  • Rule-based evaluation where the model's final output or executed code result is compared against ground truth using exact match or predefined constraint checks.

Input / output format

Input: Standardized task prompts covering diverse reasoning and planning challenges (logic, spatial, order, optimization, search, math).

Output: Autonomously generated code and/or final answer, subject to rule-based exact match or constraint verification.

Scoring recipe

def score(predictions, gold):
    if predictions == gold:
        return 1.0
    if check_constraints(predictions):
        return 1.0
    return 0.0

Common pitfalls

• The optimal reasoning mode (code vs. text) is highly task-dependent and hard to predict a priori.
• LLM-generated code frequently degenerates into hard-coded, text-like scripts that lack symbolic execution utility.
• Many tasks require iterative multi-turn exploration and optimization, which standard single-turn generation fails to capture.

Evidence (verbatim from paper)

All tasks are standardized into a unified format and evaluated using rule-based criteria (e.g., exact match or constraint checks) for efficient rollout and testing.

Citation

@misc{chen2025r1codeinterpreter,
  title={R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2505.21668}
}

• arXiv: 2505.21668