Classeval Eval

classeval-eval

ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation — Du et al. (2023) (arXiv:2308.01861, 2023)

What this evaluates

Evaluates large language models' ability to generate complete Python classes with interdependent methods, rather than standalone functions. It probes long-context code reasoning, dependency modeling, and the effectiveness of holistic versus incremental generation strategies.

Datasets

• ClassEval — total 100; splits: test (100); repo https://github.com/FudanSELab/ClassEval

Metrics

• pass@1 (primary) — range: [0, 1]
  • The proportion of generated classes that successfully pass all provided unit tests (both method-level and class-level) within a 5-second execution timeout per test case.

Input / output format

Input: A structured class skeleton containing class-level information (class name, description, imports, constructor) and method-level information (method signature, functional description, parameter/return descriptions, and example I/O).

Output: A complete Python class implementation that conforms to the specified interface and passes the provided test suite.

Scoring recipe

def compute_pass_rate(predictions, test_suites):
    passed = 0
    for code, suite in zip(predictions, test_suites):
        try:
            if run_tests_with_timeout(code, suite, timeout=5.0):
                passed += 1
        except Exception:
            pass
    return passed / len(predictions)

Common pitfalls

• Models often generate independent methods instead of a cohesive class, failing class-level tests that check inter-method dependencies.
• Method-level tests verify both return values and class field states, unlike standard function benchmarks that only check returns.
• The 5-second execution timeout may cause false negatives for models generating inefficient loops or blocking I/O operations.

Evidence (verbatim from paper)

Typically, LLMs generate code snippets based on input descriptions and the correctness is verified with the provided test suite.

Citation

@misc{du2023classeval,
  title={ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation},
  author={Du et al. (2023)},
  year={2023},
  note={arXiv:2308.01861}
}

• arXiv: 2308.01861