Chinese LLM Bench Eval

chinese-llm-bench-eval

OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training — Yu et al. (2025) (arXiv:2501.08197, 2025)

What this evaluates

Evaluates Chinese large language models' world knowledge, academic understanding, and multi-dimensional alignment after pretraining or instruction fine-tuning. It probes the model's ability to follow instructions, reason across domains, and maintain safety and helpfulness standards in Chinese.

Datasets

• C-Eval — total ?; splits: test (-1)
• CMMLU — total ?; splits: test (-1)
• Alignbench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions or aligned responses across the benchmark suite. Calculated as (correct predictions / total instances) * 100.

Input / output format

Input: For C-Eval and CMMLU: 5-shot prompting with question and options. For Alignbench: instruction prompts for multi-turn dialogue and task evaluation.

Output: Deterministic text generation (temperature=0). Expected to output the correct option letter/answer for C-Eval/CMMLU, or aligned text for Alignbench.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100

Common pitfalls

• Using temperature > 0 for deterministic multiple-choice benchmarks like C-Eval/CMMLU introduces unnecessary variance and breaks reproducibility.
• Failing to strip markdown tokens during preprocessing causes models to overfit to formatting rather than semantics, skewing benchmark scores.
• Evaluating only on accuracy without multi-dimensional alignment metrics (e.g., safety, helpfulness) misses instruction-tuning quality.

Evidence (verbatim from paper)

We assessed model performance on the C-Eval (Huang et al., 2023) and CMMLU (Li et al., 2023) benchmarks, two authoritative suites for evaluating Chinese NLP models' world-knowledge and understanding abilities. During evaluation, we set the temperature to 0 (for deterministic outputs) and used a 5-shot prompting strategy to provide limited contextual examples. As illustrated in Figure 5, the Fineweb-Edu-Chinese model experienced a sharp accuracy increase around 45k steps, surpassing the baseline by a significant margin.

Citation

@misc{yu2025opencsg,
  title={OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training},
  author={Yu et al. (2025)},
  year={2025},
  note={arXiv:2501.08197}
}

• arXiv: 2501.08197