Dutch Financial Benchmark Eval

dutch-financial-benchmark-eval

A Dutch Financial Large Language Model — Noels et al. (2024) (arXiv:2410.12835, 2024)

What this evaluates

Evaluates LLMs on domain-specific financial tasks in Dutch, including sentiment analysis, named entity recognition, relation extraction, query answering, and headline classification. It also tests cross-lingual adaptability by benchmarking the Dutch model on English financial data.

Datasets

• Dutch Financial Benchmark — total 500; splits: test (500); repo https://github.com/snoels/fingeit
• English Financial Benchmark — total ?; splits: test (-1); repo https://github.com/snoels/fingeit

Metrics

• zero-shot performance (primary) — range: [0, 1]
  • Standard classification metrics (accuracy or F1) computed per task (SA, NER, RE, QA, headline classification). The exact metric variant is not specified in the text, but performance is reported comparatively across models.

Input / output format

Input: Instruction prompt specifying a financial task (e.g., sentiment classification, NER, relation extraction, QA, or headline classification) paired with a Dutch or English financial text snippet.

Output: Task-specific answer (e.g., class label, extracted entities, relation triple, or free-text answer). Raw outputs are post-processed via an independent LLM to extract and format the answer according to the instruction.

Scoring recipe

predictions = post_process_llm_output(raw_output, task_type)
if task_type in ['SA', 'headline']:
    score = accuracy(predictions, gold_labels)
elif task_type in ['NER', 'RE']:
    score = f1_score(predictions, gold_labels, mode='strict')
elif task_type == 'QA':
    score = exact_match_or_f1(predictions, gold_answers)
return score

Common pitfalls

• LLMs without task-specific fine-tuning frequently fail to follow instructions or generate outputs in the required format, necessitating an LLM-based post-processing step for reliable answer extraction.
• Cross-lingual evaluation requires careful handling of language mismatches (e.g., a Dutch model receiving English instructions), which the authors address via translation during the post-processing stage.

Evidence (verbatim from paper)

We assess the zero-shot performance of all the models against 500 randomly chosen test samples to decrease the computational cost. ... FinGEITje outperforms the other models across all tasks, indicating the effectiveness of language-specific instruction tuning for domain-specific tasks.

Citation

@misc{noels2024dutchfinancialllm,
  title={A Dutch Financial Large Language Model},
  author={Noels et al. (2024)},
  year={2024},
  note={arXiv:2410.12835}
}

• arXiv: 2410.12835