qhjqhj00 financial-exam-qa-eval
The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systems — Xie et al. (2026) (arXiv:2602.10886, 2026)
What this evaluates
Evaluates whether LLMs can perform domain-level conceptual understanding and precise financial reasoning across multilingual professional certification exams. It probes analytical rigor and regulatory knowledge integration rather than simple factual recall.
Datasets
- EFPA — total 230; splits: test (-1)
- GRFinQA — total 268; splits: test (-1)
- CFA — total 600; splits: test (-1)
- CPA — total 300; splits: test (-1)
- BBF — total ?; splits: test (-1)
- SAHM — total 873; splits: test (-1)
Metrics
accuracy(primary) — range: [0, 1]- Accuracy is defined as the proportion of correctly identified options in the test set, calculated as the number of correct predictions divided by the total number of questions.
Input / output format
Input: A stand-alone multiple-choice question Q with four candidate answers {A1, A2, A3, A4} in one of six languages (English, Chinese, Arabic, Hindi, Greek, Spanish).
Output: The model must select and output the single correct answer A* from the four options.
Scoring recipe
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, gold_labels):
if pred == gold:
correct += 1
accuracy = correct / total
Common pitfalls
- Models may rely on surface pattern recognition or factual recall instead of the required analytical rigor and conceptual reasoning.
- Language coverage is treated as a configurable dimension rather than defining separate subtasks, so results across different language subsets may not be directly comparable.
Evidence (verbatim from paper)
The models are required to output the correct answer, and the performance is measured in terms of accuracy, defined as the proportion of correctly identified options in the test set.
Citation
@misc{xie2026finmmeval,
title={The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systems},
author={Xie et al. (2026)},
year={2026},
note={arXiv:2602.10886}
}
- arXiv: 2602.10886