Competitive Coding Eval

competitive-coding-eval

OpenCodeReasoning: Advancing Data Distillation for Competitive Coding — Wasi Uddin Ahmad et al. (2025) (arXiv:2504.01943, 2025)

What this evaluates

Evaluates the ability of large language models to generate correct, executable Python solutions for competitive programming problems. It probes algorithmic reasoning, code synthesis, and adherence to problem constraints under strict time and complexity limits.

Datasets

• LiveCodeBench — total 279; splits: test (279)
• CodeContests — total ?; splits: Public (-1), Private (-1), Generated (-1), All (-1)

Metrics

• pass@1 (primary) — range: percent
  • Average pass@1 calculated by averaging 64 inference runs for LiveCodeBench and 16 runs for CodeContests. pass@1 measures the probability that at least one of the generated solutions passes all hidden test cases.

Input / output format

Input: Natural language problem description for competitive programming tasks, including constraints and examples.

Output: Python code solution (optionally with reasoning traces), generated via temperature-based nucleus sampling with a maximum length of 30,720 tokens.

Scoring recipe

def compute_pass_at_1(generated_solutions, test_cases):
    for code in generated_solutions:
        if execute_and_check(code, test_cases):
            return 1.0
    return 0.0

# For each problem:
#   solutions = sample_n_times(model, prompt, n=64 for LCB, n=16 for CC)
#   score = compute_pass_at_1(solutions, problem.test_cases)
# Final metric = mean(score across all problems) * 100

Common pitfalls

• Averaging over multiple inference runs (64 for LiveCodeBench, 16 for CodeContests) is mandatory to reduce variance, whereas baselines were evaluated with a single run.
• Execution filtering during data curation was found to harm benchmark accuracy, so models should be evaluated on raw generated code without post-hoc execution filtering.
• The metric is reported as a percentage in the tables, but pass@1 is fundamentally a probability in [0,1].

Evidence (verbatim from paper)

To mitigate performance variance inherent in single-run evaluations, we report the average pass@1 metric, calculated by averaging 64 inference runs for LiveCodeBench and 16 runs for CodeContests.

Citation

@misc{ahmad2025opencodereasoning,
  title={OpenCodeReasoning: Advancing Data Distillation for Competitive Coding},
  author={Wasi Uddin Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2504.01943}
}

• arXiv: 2504.01943