Japanese Financial Bench Eval

japanese-financial-bench-eval

Construction of a Japanese Financial Benchmark for Large Language Models — Hirano et al. (2024) (arXiv:2403.15062, 2024)

What this evaluates

Evaluates large language models on Japanese financial domain knowledge across five distinct tasks: sentiment analysis, fundamental financial knowledge, CPA auditing, and two levels of financial planner exam questions. It probes the models' ability to understand and reason over domain-specific multiple-choice questions in Japanese.

Datasets

• chabsa — total ?; splits: test (-1); repo https://github.com/pfnet-research/japanese-lm-fin-harness
• cma Basics — total ?; splits: test (-1); repo https://github.com/pfnet-research/japanese-lm-fin-harness
• cpa Audit — total ?; splits: test (-1); repo https://github.com/pfnet-research/japanese-lm-fin-harness
• fp2 — total ?; splits: test (-1); repo https://github.com/pfnet-research/japanese-lm-fin-harness
• security_sales_1 — total ?; splits: test (-1); repo https://github.com/pfnet-research/japanese-lm-fin-harness

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted choices out of total instances. Calculated as (correct predictions / total instances) * 100.

Input / output format

Input: Multiple-choice questions in Japanese, formatted with task-specific prompts and 0-4 shot examples.

Output: The model outputs the selected choice (determined by highest likelihood or earliest appearance in generation).

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• Prompt tuning (0-4 shots) was performed per task, risking in-sample leakage.
• OpenAI API models were restricted to 0-shot due to cost, creating an unfair comparison.
• Content filters on OpenAI API blocked some responses, which were counted as incorrect.

Evidence (verbatim from paper)

To answer the multiple-choice questions, the likelihoods of the choices in the context were calculated and the choice with the highest likelihood was employed as the output. For GPT3.5 and GPT-4 series, the outputs with the temperature parameter set to 0 were obtained via API, and the choice that appeared earliest in the outputs was used as the output.

Citation

@misc{hirano2024construction,
  title={Construction of a Japanese Financial Benchmark for Large Language Models},
  author={Hirano et al. (2024)},
  year={2024},
  note={arXiv:2403.15062}
}

• arXiv: 2403.15062