Chinese LLM Benchmarks Eval

chinese-llm-benchmarks-eval

COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning — Bai et al. (2024) (arXiv:2403.18058, 2024)

What this evaluates

Evaluates the knowledge, reasoning, instruction-following, and safety alignment capabilities of Chinese instruction-tuned LLMs across academic, professional, open-ended, and safety-critical domains.

Datasets

• C-Eval — total 13948; splits: test (13948)
• CMMLU — total ?; splits: test (-1)
• BELLE-EVAL — total ?; splits: test (-1)
• SafetyBench — total 11435; splits: test (11435)

Metrics

• log-likelihood (primary) — range: [0, 1]
  • For multiple-choice questions, the model selects the option with the highest log-likelihood. Accuracy is the fraction of correct predictions.
• model-based evaluation score — range: [0, 1]
  • An LLM-as-judge or automated scoring method used to assess open-ended instruction-following responses on BELLE-EVAL.

Input / output format

Input: Multiple-choice questions with options, or open-ended instruction prompts. SafetyBench uses a few-shot setting with example prompts.

Output: For MCQs: the selected answer option letter/text. For open-ended: a generated text response.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type == 'mcq':
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return correct / len(golds)
    elif task_type == 'open_ended':
        return llm_judge_score(predictions, golds)

Common pitfalls

• Using greedy decoding instead of log-likelihood selection for MCQs, which contradicts the paper's protocol and hurts C-Eval/CMMLU scores.
• BELLE-EVAL requires model-based evaluation rather than exact string matching, so standard NLP metrics will fail.
• SafetyBench is evaluated in a few-shot setting, not zero-shot, which must be replicated for fair comparison.

Evidence (verbatim from paper)

We choosing the answer option with the highest log-likelihood as the final prediction of the model. ... We employ sampling generation for generating responses to instructions and use a model-based evaluation method.

Citation

@misc{bai2024coigcqia,
  title={COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning},
  author={Bai et al. (2024)},
  year={2024},
  note={arXiv:2403.18058}
}

• arXiv: 2403.18058