Fineval Eval

fineval-eval

FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models — Zhang et al. (2023) (arXiv:2308.09975, 2023)

What this evaluates

Evaluates large language models' knowledge and reasoning capabilities in the Chinese financial domain across multiple academic subjects like Finance, Economy, Accounting, and professional Certificates. It tests performance under zero-shot, few-shot, answer-only, and chain-of-thought prompting settings.

Datasets

• FinEval — total 4661; splits: test (-1), development (-1); repo https://github.com/SUFE-AIFLM-Lab/FinEval

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.

Input / output format

Input: A multiple-choice question in Chinese, optionally preceded by a few-shot prompt containing five examples from the development split. Prompts are provided in either Answer-Only (AO) or Chain-of-Thought (CoT) formats.

Output: A single letter corresponding to the correct option (e.g., A, B, C, D) for AO evaluation. For CoT evaluation, a step-by-step reasoning explanation followed by the final answer option.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p.strip() == g.strip())
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Chain-of-thought prompting often degrades performance on FinEval because many questions test factual knowledge rather than complex reasoning, and CoT can introduce hallucination or redundant steps.
• The reported results in the main table show the best performance across four settings (zero/few-shot × AO/CoT) per model, not a single fixed setting, which can obscure setting-specific performance drops.

Evidence (verbatim from paper)

In Table 3, the average accuracy of the random baseline and 27 models in each category is displayed, along with the weighted total scores for the four categories in the "Average" column. It should be noted that the results showcase the best performance of each model, with detailed results for different prompts available in Appendix A.

Citation

@misc{zhang2023fineval,
  title={FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Models},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2308.09975}
}

• arXiv: 2308.09975