Fin Bench V2 Eval

fin-bench-v2-eval

FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models — Kytöniemi et al. (2025) (arXiv:2512.13330, 2025)

What this evaluates

Evaluates large language models on Finnish language capabilities across reading comprehension, commonsense reasoning, sentiment analysis, world knowledge, truthfulness, and alignment. It probes both multiple-choice and generative capabilities under varying prompt formulations (cloze vs. multiple-choice) and shot configurations.

Datasets

• FIN-bench-v2 — total ?; splits: test (-1); repo https://github.com/TurkuNLP/FIN-bench-v2

Metrics

• normalized accuracy (primary) — range: [0, 1]
  • Calculated as the number of correct answers divided by the total number of answer options, or via conditional likelihood comparison over predefined options for multiple-choice tasks.
• MC2 accuracy — range: [0, 1]
  • A separate metric used specifically for the TruthfulQA MC2 task variant, returning a distinct accuracy score.
• Exact Match / F1 — range: [0, 1]
  • Exact Match measures perfect string overlap with reference answers; F1 computes token-level precision and recall for generative QA tasks like SQuAD.
• BLEU / ROUGE — range: [0, 1]
  • Automated similarity metrics used to evaluate free-form text generation against reference answers, particularly for TruthfulQA.

Input / output format

Input: Finnish language prompts formatted as either Cloze Formulation (CF) or Multiple-Choice Formulation (MCF), with five distinct prompt variants per task. Context is provided in 0-shot, 1-shot, or 5-shot configurations.

Output: For multiple-choice tasks: the model computes conditional likelihoods over predefined options to identify the correct continuation. For generative tasks: free-form text output compared against reference answers.

Scoring recipe

def compute_normalized_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_gen_metrics(predictions, references):
    em = sum(1 for p, r in zip(predictions, references) if p.strip() == r.strip()) / len(references)
    f1 = token_f1(predictions, references)
    return em, f1

Common pitfalls

• Prompt sensitivity: minor wording changes across the five prompt variants can cause substantial score variance (e.g., Belebele task).
• Formulation preference: some instruction-tuned models (e.g., Poro family) degrade in MCF compared to CF, treating option lists as noise rather than helpful constraints.
• Translation artifacts: models trained on synthetic machine-translated data may artificially outperform those trained on human-authored Finnish data due to shared stylistic features.

Evidence (verbatim from paper)

For multiple-choice tasks, we report the normalized accuracy score. The only exception is the TruthfulQA MC2 task, which returns a separate metric commonly referred to as the MC2 accuracy.

Citation

@misc{kytoniemi2025finbenchv2,
  title={FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models},
  author={Kytöniemi et al. (2025)},
  year={2025},
  note={arXiv:2512.13330}
}

• arXiv: 2512.13330