Arcade Eval

arcade-eval

Natural Language to Code Generation in Interactive Data Science Notebooks — Yin et al. (2022) (arXiv:2212.09248, 2022)

What this evaluates

Evaluates large language models' ability to generate correct Python code for interactive data science notebooks, requiring multi-turn reasoning, grounded understanding of DataFrame schemas, and composition of pandas API calls based on preceding notebook context and natural language intents.

Datasets

• ARCADE — total 1082; splits: Existing Tasks (-1), New Tasks (-1)

Metrics

• pass@k (primary) — range: [0, 1]
  • The fraction of problems with at least one correct sample given a sample size k. Estimated by drawing 50 samples per problem to reduce variance.

Input / output format

Input: Preceding notebook cells (code and comments), natural language intent for the current cell, NL description of DataFrame schema (columns and example values), and reference solutions to previous problems in multi-turn settings.

Output: Python code snippet to complete the current notebook cell.

Scoring recipe

def compute_pass_at_k(predictions_per_problem, k):
    # predictions_per_problem: list of k generated code strings
    # gold: reference solution or test suite (implied by pass@k convention)
    correct = any(is_correct(pred, gold) for pred in predictions_per_problem)
    return 1.0 if correct else 0.0

# Average over all problems to get pass@k
pass_at_k_score = sum(compute_pass_at_k(probs, k) for probs in all_problems) / len(all_problems)

Common pitfalls

• Omitting the NL schema description from prompts causes significant performance drops, especially on the New Tasks split.
• Using only the immediate intent without preceding notebook context cells leads to failures in variable reference and state tracking.
• Existing Tasks scores may be inflated due to potential data leakage from popular GitHub notebooks into model training data.

Evidence (verbatim from paper)

Following prior work (Chen et al., 2021a; Austin et al., 2021), we measure model performance using the pass@k metric, defined as the fraction of problems with at least one correct sample given a sample size k. To reduce variance, we estimate pass@k (k ≤ 30) by drawing 50 samples for each problem (Chen et al., 2021a).

Citation

@misc{yin2022arcade,
  title={Natural Language to Code Generation in Interactive Data Science Notebooks},
  author={Yin et al. (2022)},
  year={2022},
  note={arXiv:2212.09248}
}

• arXiv: 2212.09248