Ceval Eval

ceval-eval

C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models — Huang et al. (2023) (arXiv:2305.08322, 2023)

What this evaluates

Evaluates Chinese foundation models' domain knowledge and reasoning capabilities across 52 academic disciplines and four difficulty levels using multiple-choice questions. It probes the models' ability to follow instructions, perform in-context learning, and generate chain-of-thought reasoning in a Chinese language context.

Datasets

• C-EVAL — total 13948; splits: val (1346), test (-1); repo https://github.com/hkust-nlp/ceval

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as the number of questions where the extracted answer matches the ground truth option divided by the total number of questions, multiplied by 100.

Input / output format

Input: A multiple-choice question in Chinese with four options (A, B, C, D). For few-shot settings, the input includes the question plus up to five exemplars from the development split.

Output: Free-form text generation. The final answer choice is extracted from the model's response using regular expressions to match the option letter.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if pred == gold:
            correct += 1
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Chain-of-thought prompting often degrades performance on subjects that are not reasoning-intensive.
• Five-shot exemplars can exceed the context window of smaller models, requiring dynamic reduction of demonstrations.
• Instruction-tuned models may suffer accuracy drops in few-shot settings if not explicitly trained on in-context examples.
• Test split labels are not publicly released, so developers must rely on the validation split for development.

Evidence (verbatim from paper)

We report the average accuracy, while a detailed breakdown of accuracy per subject is provided in Appendix F. GPT-4 is the only model that exceeds 60% average accuracy, highlighting the challenge presented by C-EVAL.

Citation

@misc{huang2023ceval,
  title={C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models},
  author={Huang et al. (2023)},
  year={2023},
  note={arXiv:2305.08322}
}

• arXiv: 2305.08322