Taco Eval

taco-eval

TACO: Topics in Algorithmic COde generation dataset — Li et al. (2023) (arXiv:2312.14852, 2023)

What this evaluates

Evaluates code generation models on competition-level algorithmic programming problems. It probes fine-grained capabilities across different programming skills and difficulty levels by measuring whether generated Python programs correctly solve given problems under strict constraints.

Datasets

• TACO — total 26443; splits: train (-1), test (200); repo https://github.com/FlagOpen/TACO

Metrics

• pass@k (primary) — range: [0, 1]
  • Measures the probability that at least one of k generated solutions passes all provided test cases. Computed as 1 - (C(n-k, c) / C(n, c)) where n is total attempts and c is correct attempts.

Input / output format

Input: Natural language description of a programming problem, including constraints and test cases.

Output: Python 3 code block containing a complete program to solve the problem.

Scoring recipe

def pass_at_k(generated_programs, test_cases, k):
    passed = 0
    for prog in generated_programs[:k]:
        if run_all_tests(prog, test_cases):
            passed = 1
            break
    return passed
# Metric = mean(pass_at_k(prog_list, tests, k) for each problem)

Common pitfalls

• Generation hyperparameters (temperature/top-p) must be tuned per difficulty level; fixed settings severely penalize harder problems.
• GPT-4 evaluation requires a specific prompt suffix and markdown code-block extraction, differing from open-weight model pipelines.
• Generated code undergoes a common truncation post-processing step before execution, which can silently drop valid solutions if not replicated.

Evidence (verbatim from paper)

Similar to the evaluations of the generation of standard code, for each programming problem, we allow the model to generate 200 attempts, using pass@k (k=1, 10, 100) as the evaluation metric.

Citation

@misc{li2023taco,
  title={TACO: Topics in Algorithmic COde generation dataset},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2312.14852}
}

• arXiv: 2312.14852