Cfinbench Eval

cfinbench-eval

CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models — Nie et al. (2024) (arXiv:2407.02301, 2024)

What this evaluates

Evaluates large language models' domain-specific knowledge and reasoning in the Chinese financial context, covering foundational knowledge, professional certifications, practical tasks, and regulatory compliance.

Datasets

• CFinBench — total 99100; splits: development (-1), validation (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Exact match accuracy per question type. Single-choice: 1 if predicted option matches gold, else 0. Multiple-choice: 1 if predicted set exactly matches gold set, else 0. Judgment: 1 if predicted matches gold, else 0. Final score is weighted: 0.4 * single_acc + 0.4 * multiple_acc + 0.2 * judgment_acc.

Input / output format

Input: Chinese-language multiple-choice, multiple-answer, or true/false questions, optionally preceded by 3 few-shot examples.

Output: A single predicted option or answer string (answer-only setting).

Scoring recipe

def compute_cfinbench_acc(preds, golds, types):
    single_s, multi_s, judg_s = [], [], []
    for p, g, t in zip(preds, golds, types):
        if t == 'single':
            single_s.append(1.0 if p == g else 0.0)
        elif t == 'multiple':
            multi_s.append(1.0 if set(p) == set(g) else 0.0)
        else:
            judg_s.append(1.0 if p == g else 0.0)
    s_acc = sum(single_s)/len(single_s) if single_s else 0
    m_acc = sum(multi_s)/len(multi_s) if multi_s else 0
    j_acc = sum(judg_s)/len(judg_s) if judg_s else 0
    return 0.4 * s_acc + 0.4 * m_acc + 0.2 * j_acc

Common pitfalls

• Greedy decoding is used with temperature=1.0 and top_p=1.0, which is atypical for generative tasks.
• Multiple-choice scoring is strict: predicting any option outside the gold set immediately yields 0, even if correct options are included.
• Input prompts exceeding 2048 tokens are right-truncated, potentially dropping instructions or context.

Evidence (verbatim from paper)

We adopt accuracy to measure the match between model prediction and gold answer. Specifically, for single-choice questions, if multiple valid options are predicted by the model, we only select the first option as the final answer predicted by the model. For multiple-choice questions, if any of the options predicted by the model are not among the gold answer, we directly classify it as wrong. Otherwise, we score it based on the number of predicted answers (out of a full score of 1). At last, we calculate the final score for each category based on: $final=0.4\times single+0.4\times multiple+0.2\times judgment$.

Citation

@misc{nie2024cfinbench,
  title={CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models},
  author={Nie et al. (2024)},
  year={2024},
  note={arXiv:2407.02301}
}

• arXiv: 2407.02301