Clue Eval

clue-eval

CLUE: A Chinese Language Understanding Evaluation Benchmark — Xu et al. (2020) (arXiv:2004.05986, 2020)

What this evaluates

Evaluates Chinese language understanding across nine diverse tasks, including text classification, natural language inference, semantic similarity, and machine reading comprehension. It probes a model's ability to handle Chinese-specific linguistic phenomena, whole-word masking, and token-level vs. global understanding through a standardized fine-tuning pipeline.

Datasets

• CLUE — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted class labels out of the total number of instances. Computed as the number of matches divided by the total number of test samples.
• Exact Match (EM) — range: [0, 1]
  • A binary score indicating whether the predicted answer span exactly matches the gold answer span character-for-character. Used exclusively for the CMRC 2018 machine reading comprehension task.

Input / output format

Input: Single sentences for classification tasks, sentence pairs for NLI/semantic similarity, and context-question pairs for machine reading comprehension (span extraction or multiple-choice).

Output: Class labels for classification/NLI tasks, start and end token indices for span-extraction MRC, or a selected option index for multiple-choice MRC.

Scoring recipe

def compute_metric(predictions, golds, task_name):
    if task_name == 'CMRC':
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    else:
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)

Common pitfalls

• Using English-optimized subword tokenizers (e.g., SentencePiece) on Chinese text severely degrades performance on token-level tasks like span extraction.
• Assuming human performance is a reliable upper bound; some tasks (e.g., TNEWS, C3) are actually harder for humans than for machines due to ambiguous labels or complex reasoning.
• Averaging scores across all tasks without considering varying class distributions or task difficulties can mask significant performance disparities.

Evidence (verbatim from paper)

We report EM for CMRC 2018 and accuracy for all other tasks.

Citation

@misc{xu2020clue,
  title={CLUE: A Chinese Language Understanding Evaluation Benchmark},
  author={Xu et al. (2020)},
  year={2020},
  note={arXiv:2004.05986}
}

• arXiv: 2004.05986