Cuge Eval

cuge-eval

CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark — Yao et al. (2021) (arXiv:2112.13610, 2021)

What this evaluates

Evaluates Chinese language understanding and generation capabilities across a hierarchical framework. It probes discourse comprehension, conversational interaction, mathematical reasoning, and multilingual tasks using a multi-level scoring strategy that normalizes model performance against a fixed baseline.

Datasets

• CUGE (lite version) — total ?; splits: test (-1)

Metrics

• normalized capability performance (primary) — range: [0, 1]
  • Raw model performance (e.g., accuracy) normalized against the mT5-Small baseline score on the same task. The exact normalization function is not specified, but it scales model results relative to the 300M-parameter baseline to enable cross-capability comparison.

Input / output format

Input: Chinese language prompts/tasks spanning understanding and generation capabilities (e.g., discourse, conversation, math, summarization).

Output: Model-generated responses or predicted labels for each Chinese NLP task instance.

Scoring recipe

def compute_normalized_score(model_preds, baseline_preds, gold):
    model_acc = accuracy(model_preds, gold)
    baseline_acc = accuracy(baseline_preds, gold)
    # Normalize relative to baseline performance
    normalized = (model_acc - baseline_acc) / (1.0 - baseline_acc)
    return normalized

Common pitfalls

• Uses a 'lite version' of CUGE for evaluation rather than the full benchmark suite.
• Reports normalized scores relative to mT5-Small, which can mask absolute performance gaps or ceiling effects.
• Aggregates metrics across capabilities, tasks, and datasets, potentially hiding weaknesses in specific sub-tasks.

Evidence (verbatim from paper)

In our experiments, we select mT5-SmallXue et al. (2020), a representative pre-trained language model with $300$M parameters as our standard baseline model to normalize the performance of models under evaluation. ... We report the experimental results in Table2 and visualize the normalized capability performance in Figure3

Citation

@misc{yao2021cuge,
  title={CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark},
  author={Yao et al. (2021)},
  year={2021},
  note={arXiv:2112.13610}
}

• arXiv: 2112.13610