Fcmbench Eval

fcmbench-eval

FCMBench: A Comprehensive Financial Credit Multimodal Benchmark for Real-world Applications — Yang et al. (2026) (arXiv:2601.00150, 2026)

What this evaluates

Evaluates vision-language models on financial credit document understanding, covering perception tasks like document type recognition and key information extraction, as well as reasoning tasks such as validity checking and numerical calculation under strict low-latency constraints.

Datasets

• FCMBench — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: percent
  • Computed per task (DTR, KIE, IQE, CC, VC, NC, RR) and averaged across all tasks. For KIE, field-level exactness is required after normalization rather than semantic similarity. Standard token-level or exact-match F1 is used for other tasks.

Input / output format

Input: Financial document images paired with task-specific text prompts.

Output: Text predictions containing labels, extracted field values, or reasoning steps.

Scoring recipe

def compute_fcmbench_f1(predictions, golds):
    task_scores = []
    for task in ['DTR', 'KIE', 'IQE', 'CC', 'VC', 'NC', 'RR']:
        preds_t = [p for p, g in zip(predictions, golds) if g.task == task]
        golds_t = [g for p, g in zip(predictions, golds) if g.task == task]
        if task == 'KIE':
            correct = sum(1 for p, g in zip(preds_t, golds_t) if normalize(p) == normalize(g))
            f1 = correct / len(golds_t) if golds_t else 0
        else:
            f1 = compute_f1(preds_t, golds_t)
        task_scores.append(f1)
    return sum(task_scores) / len(task_scores)

Common pitfalls

• Commercial models with mandatory 'Think' mode are compared against instruct-only models, creating an unfair advantage on reasoning tasks.
• KIE scoring requires strict field-level exactness after normalization, not semantic similarity, which differs from standard LLM evaluation protocols.
• Evaluation prioritizes low-latency 'Instruct' mode to mimic real credit approval pipelines, so results do not reflect theoretical upper bounds with unrestricted chain-of-thought reasoning.

Evidence (verbatim from paper)

As shown in Figure 6(a), the F1 scores of the tested models range from approximately 30 to 65, with an average F1 score of 45.9% ± 9.2% . This indicates that FCMBench is a challenging benchmark and can effectively distinguish performance differences among VLMs.

Citation

@misc{yang2026fcmbench,
  title={FCMBench: A Comprehensive Financial Credit Multimodal Benchmark for Real-world Applications},
  author={Yang et al. (2026)},
  year={2026},
  note={arXiv:2601.00150}
}

• arXiv: 2601.00150