Financemath Eval

financemath-eval

FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains — Zhao et al. (2023) (arXiv:2311.09797, 2023)

What this evaluates

This benchmark evaluates large language models' ability to perform knowledge-intensive mathematical reasoning within the finance domain. It requires models to integrate college-level financial knowledge with both textual descriptions and tabular data to solve complex problems.

Datasets

• FinanceMATH — total 1200; splits: test (-1), development (-1)

Metrics

• accuracy (primary) — range: percent
  • The proportion of instances where the final answer extracted from the model's output exactly matches the ground-truth answer.

Input / output format

Input: Math reasoning problems containing textual descriptions and tabular data, with tables serialized in Markdown format.

Output: For Chain-of-Thought (CoT): a natural language reasoning trace ending with a final answer. For Program-of-Thought (PoT): a Python solution script that is executed to compute the final answer.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        # Extract final answer from CoT text or execute PoT Python code
        extracted_answer = extract_final_answer(pred)
        if extracted_answer == gold:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• For PoT prompting, the generated Python code must be syntactically correct and executable; execution failures prevent answer extraction.
• CoT answer extraction relies on a specific pipeline (Chen et al., 2023c) to correctly parse the final answer from verbose reasoning traces.
• Models must correctly interpret tabular data serialized in Markdown format, which can lead to misinterpretation of required domain knowledge.

Evidence (verbatim from paper)

Once we obtain the final answer from model's output, we compare it with the ground-truth answer for accuracy measurement.

Citation

@misc{zhao2023financemath,
  title={FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains},
  author={Zhao et al. (2023)},
  year={2023},
  note={arXiv:2311.09797}
}

• arXiv: 2311.09797