# Finchain Eval

> Evaluates multi-step symbolic financial reasoning by measuring how well language models generate verifiable chain-of-thought traces aligned with executable financial templates. It probes both the semantic and numeric consistency of intermediate reasoning steps and the accuracy of the final financial answer. Use when the user wants to benchmark on FinChain, or asks about evaluating this task. Reports ChainEval.

- Skill: `qhjqhj00/finchain-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/finchain-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/finchain-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/finchain-eval

---


# finchain-eval

> FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning — Xie et al. (2025) (arXiv:2506.02515, 2025)

## What this evaluates

Evaluates multi-step symbolic financial reasoning by measuring how well language models generate verifiable chain-of-thought traces aligned with executable financial templates. It probes both the semantic and numeric consistency of intermediate reasoning steps and the accuracy of the final financial answer.

## Datasets

- **FinChain** — total 2900; splits: test (2900); repo https://github.com/mbzuai-nlp/finchain

## Metrics

- `ChainEval` **(primary)** — range: percent
  - A dynamic time warping (DTW) based normalized gate metric that jointly measures final-answer correctness and alignment of intermediate reasoning steps, capturing both semantic and numeric consistency across steps.
- `FAC` — range: percent
  - Final Answer Correctness, measuring whether the model's final numerical or categorical answer matches the gold standard within a specified tolerance.
- `ROUGE-2` — range: percent
  - Bigram overlap between predicted and reference reasoning traces.
- `ROUGE-L` — range: percent
  - Longest common subsequence overlap between predicted and reference reasoning traces.
- `BERTScore` — range: [0, 1]
  - Contextual embedding similarity between predicted and reference reasoning traces.

## Input / output format

**Input**: A standardized zero-shot prompt: 'Please answer the given question and provide a step-by-step solution. Use the format: Step 1: ..., Step 2: ..., ... The question is: {q}'

**Output**: A step-by-step solution following the 'Step x: ...' format, concluding with a final answer.

## Scoring recipe

```python
# Extract steps from model output using regex to handle formatting variations
predicted_steps = extract_regex_steps(model_output, pattern=r"step\s*\d*[:\s]*")
gold_steps = gold_trace.steps

# Compute DTWNormGate alignment score between predicted and gold steps
alignment_score = compute_dtw_norm_gate(predicted_steps, gold_steps)

# Check final answer correctness (FAC) within tolerance
fac_score = 1.0 if is_answer_correct(model_output.final_answer, gold_answer, tolerance) else 0.0

# ChainEval combines alignment and FAC
chaineval_score = alignment_score * fac_score
return chaineval_score, fac_score
```

## Common pitfalls

- Strict string matching fails due to formatting variations; regex post-processing is required to reliably align reasoning steps.
- Surface-level metrics (ROUGE, BERTScore) correlate poorly with step-level reasoning fidelity and human expert judgments.
- Numerical deviations in intermediate steps may be deemed immaterial by experts, requiring tolerance-based evaluation rather than exact match.

## Evidence (verbatim from paper)

> We use ChainEval as the primary evaluation measure, as it jointly measures final-answer correctness and alignment of the intermediate reasoning steps. We post-processed model outputs with regular expressions to extract the ordered list of reasoning steps, accommodating common variations such as “Step x:”, “Step x”, or “stepx”.

## Citation

```bibtex
@misc{xie2025finchain,
  title={FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning},
  author={Xie et al. (2025)},
  year={2025},
  note={arXiv:2506.02515}
}
```

- arXiv: 2506.02515