Cci30 Hq Eval

cci30-hq-eval

CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models — Wang et al. (2024) (arXiv:2410.18505, 2024)

What this evaluates

Evaluates the effectiveness of a high-quality Chinese pre-training corpus by training a 0.5B language model from scratch and measuring its zero-shot generalization across standard English and Chinese knowledge benchmarks. The protocol also includes a comparative analysis of data quality classifiers using macro F1-score on a held-out test set.

Datasets

• Standard Benchmarks (ARC-C, ARC-E, HellaSwag, Winograd, MMLU, OpenbookQA, PIQA, SIQA, CEval, CMMLU) — total ?; splits: test (-1)

Metrics

• Average (primary) — range: [0, 1]
  • Arithmetic mean of zero-shot accuracy scores across 10 standard benchmarks (ARC-C, ARC-E, HellaSwag, Winograd, MMLU, OpenbookQA, PIQA, SIQA, CEval, CMMLU).

Input / output format

Input: Zero-shot natural language prompts corresponding to each benchmark task (multiple-choice or open-ended QA).

Output: Model-generated text or selected option corresponding to the ground truth answer.

Scoring recipe

def compute_average_accuracy(predictions, golds, benchmarks):
    task_scores = []
    for task in benchmarks:
        acc = sum(1 for p, g in zip(predictions[task], golds[task]) if p == g) / len(golds[task])
        task_scores.append(acc)
    return sum(task_scores) / len(task_scores)

Common pitfalls

• Benchmarks are evaluated in zero-shot mode, unlike standard few-shot protocols for tasks like MMLU.
• The 'Average' metric treats all 10 benchmarks equally despite varying difficulty and domain coverage.
• Classifier evaluation uses a fixed 0-5 score threshold of 3.0, which may not align with optimal precision-recall trade-offs across different data distributions.

Evidence (verbatim from paper)

Evaluation metrics include: Average_Chinese : Average score of Chinese metrics, including CEval and CMMLU. Average_English : Average score across standard English metrics such as ARC-C, ARC-E, HellaSwag, Winograd, MMLU, OpenbookQA, PIQA and SIQA. Average: Combined average score of all evaluation metrics above.

Citation

@misc{wang2024cci30hq,
  title={CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2410.18505}
}

• arXiv: 2410.18505