Finchain Eval

finchain-eval

FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning — Xie et al. (2025) (arXiv:2506.02515, 2025)

What this evaluates

Evaluates multi-step symbolic financial reasoning by measuring how well language models generate verifiable chain-of-thought traces aligned with executable financial templates. It probes both the semantic and numeric consistency of intermediate reasoning steps and the accuracy of the final financial answer.

Datasets

• FinChain — total 2900; splits: test (2900); repo https://github.com/mbzuai-nlp/finchain

Metrics

• ChainEval (primary) — range: percent
  • A dynamic time warping (DTW) based normalized gate metric that jointly measures final-answer correctness and alignment of intermediate reasoning steps, capturing both semantic and numeric consistency across steps.
• FAC — range: percent
  • Final Answer Correctness, measuring whether the model's final numerical or categorical answer matches the gold standard within a specified tolerance.
• ROUGE-2 — range: percent
  • Bigram overlap between predicted and reference reasoning traces.
• ROUGE-L — range: percent
  • Longest common subsequence overlap between predicted and reference reasoning traces.
• BERTScore — range: [0, 1]
  • Contextual embedding similarity between predicted and reference reasoning traces.

Input / output format

Input: A standardized zero-shot prompt: 'Please answer the given question and provide a step-by-step solution. Use the format: Step 1: ..., Step 2: ..., ... The question is: {q}'

Output: A step-by-step solution following the 'Step x: ...' format, concluding with a final answer.

Scoring recipe

# Extract steps from model output using regex to handle formatting variations
predicted_steps = extract_regex_steps(model_output, pattern=r"step\s*\d*[:\s]*")
gold_steps = gold_trace.steps

# Compute DTWNormGate alignment score between predicted and gold steps
alignment_score = compute_dtw_norm_gate(predicted_steps, gold_steps)

# Check final answer correctness (FAC) within tolerance
fac_score = 1.0 if is_answer_correct(model_output.final_answer, gold_answer, tolerance) else 0.0

# ChainEval combines alignment and FAC
chaineval_score = alignment_score * fac_score
return chaineval_score, fac_score

Common pitfalls

• Strict string matching fails due to formatting variations; regex post-processing is required to reliably align reasoning steps.
• Surface-level metrics (ROUGE, BERTScore) correlate poorly with step-level reasoning fidelity and human expert judgments.
• Numerical deviations in intermediate steps may be deemed immaterial by experts, requiring tolerance-based evaluation rather than exact match.

Evidence (verbatim from paper)

We use ChainEval as the primary evaluation measure, as it jointly measures final-answer correctness and alignment of the intermediate reasoning steps. We post-processed model outputs with regular expressions to extract the ordered list of reasoning steps, accommodating common variations such as “Step x:”, “Step x”, or “stepx”.

Citation

@misc{xie2025finchain,
  title={FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning},
  author={Xie et al. (2025)},
  year={2025},
  note={arXiv:2506.02515}
}

• arXiv: 2506.02515