Unconventional Problems Eval

unconventional-problems-eval

Textbooks Are All You Need — Gunasekar et al. (2023) (arXiv:2306.11644, 2023)

What this evaluates

Evaluates code generation on novel, synthetic programming problems designed to avoid training data contamination. It assesses the model's ability to understand and implement complex logic beyond simple unit test passing.

Datasets

• Unconventional Problems — total 50; splits: test (50)

Metrics

• LLM graded Understanding score (primary) — range: percent
  • GPT-4 grades each generated solution on a 0–10 scale based on how well it matches the expected logic, preceded by a short verbal evaluation. The final score is the average grade across all 50 problems, multiplied by 10 to report as a percentage.

Input / output format

Input: Python function signature and docstring describing the expected behavior.

Output: Python code implementing the function.

Scoring recipe

def compute_metric(predictions, gold):
    grades = []
    for pred in predictions:
        # GPT-4 prompt evaluates pred against gold logic
        # returns verbal eval + grade 0-10
        grade = call_gpt4_grader(pred, gold)
        grades.append(grade)
    return (sum(grades) / len(grades)) * 10

Common pitfalls

• Relies on GPT-4 as a grader, which may introduce subjective bias or inconsistency compared to deterministic unit tests.
• Problems were specifically designed to be 'unconventional' and outside the training distribution, so results may not generalize to standard coding benchmarks.

Evidence (verbatim from paper)

To evaluate candidate solutions, we therefore adopt the approach of using GPT-4 to grade the solution (such as in [EL23]). This approach has two distinct advantages: (1) by using GPT-4 as a grader, we can leverage its knowledge and generative abilities to obtain a more fine-grained and meaningful signal of the student model's coding capabilities, and (2) it obviates the need for tests. Our prompt instructs the LLM to evaluate a student's solution first in a short verbal evaluation followed by grades from 0 to 10. Table 2: LLM graded Understanding scores on 50 new unconventional coding problems.

Citation

@misc{gunasekar2023textbooks,
  title={Textbooks Are All You Need},
  author={Gunasekar et al. (2023)},
  year={2023},
  note={arXiv:2306.11644}
}

• arXiv: 2306.11644