Cgce Eval

cgce-eval

CGCE: A Chinese Generative Chat Evaluation Benchmark for General and Financial Domains — Zhang et al. (2023) (arXiv:2305.14471, 2023)

What this evaluates

Evaluates Chinese generative chat models on general knowledge and financial domain tasks, measuring response quality across multiple human-assessed dimensions. It probes the model's ability to handle diverse prompts in mathematics, reasoning, scenario writing, and financial analysis, while assessing the overall quality of the generated Chinese text.

Datasets

• CGCE — total 350; splits: test (350)

Metrics

• accuracy (primary) — range: other
  • Human annotators rate the factual correctness and relevance of the model's response to the prompt.
• coherence — range: other
  • Human annotators rate the logical flow and structural consistency of the generated text.
• clarity — range: other
  • Human annotators rate how clearly and understandably the response conveys its points.
• completeness — range: other
  • Human annotators rate whether the response fully addresses all aspects of the prompt.

Input / output format

Input: Chinese-language prompts/questions spanning general domains (math, reasoning, scenario writing) and financial domains (terminology, market commentary, data analysis, news).

Output: Free-form Chinese text responses generated by the model.

Scoring recipe

def compute_cgce_score(predictions, golds):
    total_score = 0.0
    for pred, gold in zip(predictions, golds):
        acc = human_annotate(pred, 'accuracy')
        coh = human_annotate(pred, 'coherence')
        clr = human_annotate(pred, 'clarity')
        comp = human_annotate(pred, 'completeness')
        total_score += (acc + coh + clr + comp) / 4.0
    return total_score / len(predictions)

Common pitfalls

• Scoring relies entirely on human judgment rather than automated metrics, introducing inter-annotator variability.
• The benchmark is exclusively in Chinese, making it inapplicable to non-Chinese models or cross-lingual evaluations.
• No standardized rubric or scoring scale (e.g., 1-5) is explicitly defined in the provided text, requiring external documentation for exact thresholds.

Evidence (verbatim from paper)

The benchmark employs multi-dimensional human scoring on accuracy, coherence, clarity, and completeness to assess model performance in real-world Chinese generative tasks, addressing the critical gap in domain-specific and language-specific evaluation for Chinese chat models.

Citation

@misc{zhang2023cgce,
  title={CGCE: A Chinese Generative Chat Evaluation Benchmark for General and Financial Domains},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2305.14471}
}

• arXiv: 2305.14471