Dianjin R1 Eval

dianjin-r1-eval

DianJin-R1: Evaluating and Enhancing Financial Reasoning in Large Language Models — Jie Zhu et al. (2025) (arXiv:2504.15716, 2025)

What this evaluates

Evaluates large language models' financial reasoning capabilities and general problem-solving skills across multiple benchmarks. It measures how well models can answer domain-specific financial questions and general math/science reasoning tasks, while also assessing compliance rule adherence in Chinese financial contexts.

Datasets

• CFLUE — total 3864; splits: test (3864)
• FinQA — total 1147; splits: test (1147)
• CCC — total 200; splits: test (200)
• MATH-500 — total 500; splits: test (500)
• GPQA-Diamond — total 198; splits: test (198)

Metrics

• accuracy (primary) — range: percent
  • The proportion of correctly answered questions out of the total number of test instances. Average accuracy is computed across all five test sets.

Input / output format

Input: Natural language financial reasoning questions or general math/science problems, provided in either Chinese or English.

Output: Predicted answer string. For multiple-choice questions, the predicted option; for open-ended questions, the extracted final answer or compliance judgment.

Scoring recipe

def compute_accuracy(predictions, golds, dataset_name):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if dataset_name in ['FinQA', 'CCC']:
            is_correct = gpt4o_evaluate(pred, gold, prompt_file)
        else:
            extracted_pred = extract_answer(pred)
            is_correct = (extracted_pred == gold)
        correct += is_correct
    return (correct / len(predictions)) * 100

Common pitfalls

• Evaluation uses different methods per dataset: rule-based exact matching for CFLUE/MATH/GPQA, but GPT-4o LLM-as-a-judge for FinQA and CCC, which can introduce inconsistency.
• CCC is a proprietary in-house dataset and not publicly available, limiting direct replication of results.
• RL training data was exclusively Chinese (from CFLUE), which caused a performance drop on the English FinQA benchmark due to language mismatch.

Evidence (verbatim from paper)

For each dataset, we report the accuracy—defined as the proportion of correctly answered questions—and compute the average accuracy across all test sets. Among them, CFLUE and CCC are Chinese-language datasets, while the others are in English. The detailed statistics of these test sets are summarized in Table 2. For FinQA and CCC, we use GPT-4o to evaluate the correctness of each answer, following the prompts shown in Figure 8 and Figure 9 in Appendix B. For the other test sets, we extract the predicted answers using rule-based methods and compare them directly with the gold answers.

Citation

@misc{zhu2025dianjinr1,
  title={DianJin-R1: Evaluating and Enhancing Financial Reasoning in Large Language Models},
  author={Jie Zhu et al. (2025)},
  year={2025},
  note={arXiv:2504.15716}
}

• arXiv: 2504.15716