Fin Bench Eval

fin-bench-eval

FinGPT: Large Generative Models for a Small Language — Luukkonen et al. (2023) (arXiv:2311.05640, 2023)

What this evaluates

Evaluates Finnish large language models on a curated suite of 11 tasks spanning arithmetic, reasoning, general knowledge, emotion classification, and linguistic understanding. It probes few-shot generalization and task-specific capabilities in a low-resource language setting.

Datasets

• FIN-bench — total 3919; splits: test (3919)

Metrics

• mean accuracy (primary) — range: [0, 1]
  • Accuracy is computed per task. For tasks with subtasks (Cause and effect, Arithmetic), accuracy is first averaged over subtasks, then averaged across all tasks to produce a single mean accuracy score.

Input / output format

Input: Natural language questions or prompts in Finnish, presented in zero- to three-shot in-context learning format.

Output: Model-generated text responses, typically a direct answer or selected option corresponding to the prompt.

Scoring recipe

task_accuracies = []
for task in tasks:
    if task.has_subtasks:
        subtask_accs = [accuracy(task.subtask_i) for i in subtasks]
        task_acc = mean(subtask_accs)
    else:
        task_acc = accuracy(task)
    task_accuracies.append(task_acc)
return mean(task_accuracies)

Common pitfalls

• Small models may show better zero-shot than few-shot performance due to pattern copying from preceding context rather than genuine reasoning.
• Performance can drop unexpectedly between 8B and 13B models due to overfitting on limited non-repeated text, not just scaling laws.
• Subtask accuracies must be averaged before computing the overall mean to prevent tasks with more examples from dominating the final score.

Evidence (verbatim from paper)

We evaluate models on FIN-bench in zero- to three-shot settings and summarize results using mean accuracy across all tasks. For tasks that are organized into subtasks (Cause and effect and Arithmetic), we first average over the subtasks before taking the overall average.

Citation

@misc{luukkonen2023fingpt,
  title={FinGPT: Large Generative Models for a Small Language},
  author={Luukkonen et al. (2023)},
  year={2023},
  note={arXiv:2311.05640}
}

• arXiv: 2311.05640