Cflue Eval

cflue-eval

Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Dataset — Jie Zhu et al. (2024) (arXiv:2405.10542, 2024)

What this evaluates

Evaluates large language models' proficiency in Chinese financial domain knowledge and their ability to perform standard NLP tasks within the financial sector. It probes both factual recall and reasoning via multiple-choice qualification exams, as well as practical application skills like text classification, machine translation, relation extraction, reading comprehension, and text generation.

Datasets

• CFLUE — total ?; splits: train (30908), valid (3864), test (20386); repo https://github.com/aliyun/cflue

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact-match accuracy calculated as the number of correctly predicted instances divided by the total number of instances. For multiple-choice questions, the predicted option must exactly match the ground truth option.

Input / output format

Input: Task-specific prompt containing instructions, source text/documents (if applicable), and question/options. For knowledge assessment: multiple-choice question with 2-6 options. For other tasks: raw financial text or document with task-specific query.

Output: For knowledge: selected option letter or text. For classification: predicted class label. For MT/RC/TG: generated text string. For RE: structured entity/relation tuples.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for p, g in zip(predictions, golds):
        if normalize(p) == normalize(g):
            correct += 1
    return correct / len(golds)

Common pitfalls

• 60% of knowledge questions have up to 6 choices, 10% are true-false, and the rest allow multiple correct answers; scorers must handle single vs. multiple correct options correctly.
• Application tasks are test-only with no validation set provided; models must be evaluated zero-shot or with fixed few-shot prompts without hyperparameter tuning.
• Questions were rephrased and choices shuffled by GPT-4 to mitigate contamination; evaluators must use the exact provided versions, not original exam sources.

Evidence (verbatim from paper)

revealing that only GPT-4 and GPT-4-turbo exceed 60% accuracy in knowledge assessment, while lightweight LLMs like ChatGLM3-6B and Qwen-7B achieve competitive performance with 4% fewer parameters through supervised fine-tuning.

Citation

@misc{zhu2024cflue,
  title={Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Dataset},
  author={Jie Zhu et al. (2024)},
  year={2024},
  note={arXiv:2405.10542}
}

• arXiv: 2405.10542