Bizfinbench Eval

bizfinbench-eval

BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs — Lu et al. (2025) (arXiv:2505.19457, 2025)

What this evaluates

Evaluates LLMs on real-world financial reasoning tasks, including numerical calculation, temporal reasoning, information extraction, prediction recognition, and knowledge-based QA in Chinese. It probes the models' ability to handle noisy, context-dependent financial data and produce structured, reasoned outputs.

Datasets

• BizFinBench — total 6781; splits: test (6781); repo https://github.com/HiThink-Research/BizFinBench

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered queries per task, calculated as (correct predictions / total predictions) * 100. Scores are reported per task and averaged across all nine tasks.

Input / output format

Input: Chinese-language financial queries/tasks spanning nine categories: AEA, FNC, FTR, FTU, FQA, FDD, ER, SP, and FNER.

Output: Strictly formatted JSON containing two mandatory fields: 'cot' (detailed chain-of-thought logic trace) and 'Answer' (final conclusion).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        model_answer = json.loads(pred)['Answer']
        if model_answer.strip().lower() == gold_ans.strip().lower():
            correct += 1
    return (correct / len(gold)) * 100

Common pitfalls

• Models must output strictly valid JSON with exact field names ('cot' and 'Answer'); parsing failures are common if formatting constraints are ignored.
• Evaluation uses GPT-4o as a unified judge, which may introduce bias or inconsistency; the paper introduces IteraJudge to mitigate this, but standard evaluation relies on the judge's correlation with human/expert labels.
• Tasks are in Chinese and require domain-specific financial knowledge; models trained primarily on English data may underperform significantly.

Evidence (verbatim from paper)

All LLMs were configured with a maximum generation length of 1,024 tokens, temperature parameter T=0, and batch size B=1000. We employed GPT-4o as the unified evaluation judge. ... we constrained all models to produce strictly JSON-formatted responses containing two mandatory fields: ① Chain-of-Thought(cot): Detailed logic trace with intermediate steps; and ② Answer: Final conclusion derived after reasoning.

Citation

@misc{lu2025bizfinbench,
  title={BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs},
  author={Lu et al. (2025)},
  year={2025},
  note={arXiv:2505.19457}
}

• arXiv: 2505.19457