Ds 1000 Eval

ds-1000-eval

DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation — Lai et al. (2022) (arXiv:2211.11501, 2022)

What this evaluates

This benchmark evaluates a model's ability to generate correct, executable Python code for data science tasks, specifically focusing on NumPy operations. It probes functional correctness under natural language descriptions and tests robustness against surface-form and semantic perturbations of original StackOverflow problems.

Datasets

• numpy-100 — total 100; splits: test (100)

Metrics

• pass@1 (primary) — range: [0, 1]
  • Fraction of problems where at least one of the generated samples passes all provided test cases. Computed over sampled generations per problem.

Input / output format

Input: Natural language problem description followed by a code template containing an [insert] placeholder, e.g., Problem: <description>\n<code>\nimport numpy as np\n[insert]\nprint(result)\n</code>

Output: Python code snippet that fills the [insert] placeholder to solve the problem.

Scoring recipe

def compute_pass_at_1(generated_codes, test_cases):
    passed_count = 0
    for code in generated_codes:
        try:
            exec(code, {"np": __import__("numpy")})
            if all(check_test_case(code, tc) for tc in test_cases):
                passed_count += 1
                break
        except Exception:
            pass
    return passed_count / len(generated_codes)

Common pitfalls

• Models may memorize original StackOverflow solutions; perturbations are required to measure true generalization.
• Execution-based evaluation requires a secure sandbox; models might attempt to read/write files or access the network, which should be blocked.
• Surface-form constraints (e.g., specific API usage) are often overlooked if only functional test cases are checked.

Evidence (verbatim from paper)

At last, we equipped each problem and its perturbation with one test case and an automatic evaluation. Then we tested the performance of Codex-002 on them. We sampled 20 problems from numpy-100 and generated 10 samples for each problem with temperature set to 0.7, and top-p cutoff set to 0.95.

Citation

@misc{lai2022ds1000,
  title={DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation},
  author={Lai et al. (2022)},
  year={2022},
  note={arXiv:2211.11501}
}

• arXiv: 2211.11501