Multifinben Eval

multifinben-eval

MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application — Peng et al. (2025) (arXiv:2506.14028, 2025)

What this evaluates

Evaluates large language models on financial reasoning, comprehension, and generation across multiple modalities (text, vision, audio), languages (English, Chinese, Japanese, Spanish, Greek), and task types (IE, QA, summarization, etc.), using a difficulty-aware selection framework to ensure balanced and discriminative assessment.

Datasets

• IESC — total 8630; splits: test (8630)
• FinRED — total 1068; splits: test (1068)
• FINER-ORD — total 1080; splits: test (1080)
• Headlines — total 2283; splits: test (2283)
• TATSA — total 561; splits: test (561)
• BRL-Math — total 1000; splits: test (1000)
• FinQA — total 1147; splits: test (1147)
• TATQA — total 1668; splits: test (1668)
• CECTSUM — total 495; splits: test (495)
• TGEDTSUM — total 2000; splits: test (2000)
• RMCCF — total 2278; splits: test (2278)
• BigData22 — total 1470; splits: test (1470)
• MDSFT — total 222; splits: test (222)
• RRE — total 117; splits: test (117)
• AIE — total 1573; splits: test (1573)
• LNE — total 218; splits: test (218)
• FinanceIQ — total 7123; splits: test (7123)
• chabsa — total 7723; splits: test (7723)
• MultiFin — total 368; splits: test (368)
• EFPA — total 50; splits: test (50)
• FNS-2023 — total 50; splits: test (50)
• GRFinNUM — total 100; splits: test (100)
• GRMultiFin — total 54; splits: test (54)
• GRFinQA — total 225; splits: test (225)
• GRFNS-2023 — total 50; splits: test (50)
• DOLFIN — total 1932; splits: test (1932)
• PolyFiQA-Easy — total 7172; splits: test (7172)
• PolyFiQA-Expert — total 7172; splits: test (7172)
• EnglishOCR — total 17961; splits: test (17961)
• JapaneseOCR — total 217586; splits: test (217586)
• SpanishOCR — total 312819; splits: test (312819)
• GreekOCR — total 46533; splits: test (46533)
• TableBench — total 450; splits: test (450)
• MDRM-test — total 22208; splits: test (22208)
• FinAudioSum — total 64; splits: test (64)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted answers or classifications out of the total number of instances.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall, typically computed as macro or micro average depending on the task.
• ROUGE-1 — range: [0, 1]
  • Unigram overlap between generated and reference text, calculated as 2 * (precision * recall) / (precision + recall).
• WER — range: [0, 1]
  • Word Error Rate, measuring the number of insertions, deletions, and substitutions divided by the total number of words in the reference.

Input / output format

Input: Task-specific inputs including financial text, scanned documents/images, audio recordings, and multilingual prompts requiring cross-lingual evidence integration.

Output: Task-specific outputs: extracted entities/relations, multiple-choice answers, generated text/summaries, or structured financial data.

Scoring recipe

def compute_metric(predictions, gold, metric_name):
    if metric_name == 'Accuracy':
        return sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    elif metric_name == 'F1':
        return f1_score(gold, predictions, average='macro')
    elif metric_name == 'ROUGE-1':
        return rouge_score(gold, predictions, rouge_types=['rouge1'])
    elif metric_name == 'WER':
        return wer_score(gold, predictions)
    return 0.0

Common pitfalls

• Difficulty calibration relies on GPT-4o and LLaMA-3.1-70B-Instruct, which may not reflect all model capabilities or introduce bias toward their training data.
• Cross-lingual and multimodal tasks require careful handling of modality alignment and language-specific financial terminology, which can lead to inconsistent performance across languages.
• ROUGE metrics may not capture factual accuracy or financial reasoning quality in QA and generation tasks.

Evidence (verbatim from paper)

QAFinQA *Chen et al. ([2021])*Annual Reports1,147AccuracyMIT LicenseHard

Citation

@misc{peng2025multifinben,
  title={MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application},
  author={Peng et al. (2025)},
  year={2025},
  note={arXiv:2506.14028}
}

• arXiv: 2506.14028