Leetcode Dataset Eval

leetcode-dataset-eval

LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs — Xia et al. (2025) (arXiv:2504.14655, 2025)

What this evaluates

Evaluates code generation and algorithmic reasoning capabilities on competitive programming problems. It specifically probes temporal robustness by testing on problems released after a strict cutoff date to detect data contamination, and measures performance across difficulty levels and algorithmic topics.

Datasets

• LeetCodeDataset — total 256; splits: test (256); repo https://github.com/newfacade/LeetCodeDataset

Metrics

• pass@1 (primary) — range: percent
  • The proportion of problems where the model's single generated solution passes all hidden test cases. Calculated as (number of correctly solved problems) / (total number of problems).

Input / output format

Input: Programming problem description including statement, constraints, and example I/O.

Output: A single Python code solution for the given problem.

Scoring recipe

def compute_pass_at_1(predictions, gold_tests):
    passed = 0
    for pred, tests in zip(predictions, gold_tests):
        if run_tests(pred, tests):  # Executes code against all test cases
            passed += 1
    return (passed / len(predictions)) * 100

Common pitfalls

• Using generation parameters other than temperature=0.2 and top_p=0.95 yields non-comparable pass rates.
• Assuming temporal split guarantees zero contamination without verifying exact problem release dates against model training cutoffs.
• Aggregating pass rates across difficulty levels or topics without weighting by problem count misrepresents overall performance.

Evidence (verbatim from paper)

We evaluate six models on the LeetCodeDataset test set, comprising 256 programming problems that were newly released after July 1, 2024. The evaluation highlights DeepSeek-R1 (pass@1 rate = 65.23%) and QwQ-Plus (pass@1 rate = 56.25%) as top performers, demonstrating the substantial advantage of long-CoT reasoning models in solving complex competition-level coding problems. Following LiveCodeBench’s temporal evaluation methodology, we analyze monthly accuracy change relative to problem release months as shown in [Figure 3], and summarize model pass rates across difficulty levels in [Table 2].

Citation

@misc{xia2025leetcode,
  title={LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs},
  author={Xia et al. (2025)},
  year={2025},
  note={arXiv:2504.14655}
}

• arXiv: 2504.14655