Dutch LLM Bench Eval

dutch-llm-bench-eval

Fietje: An open, efficient LLM for Dutch — Vanroy et al. (2024) (arXiv:2412.15450, 2024)

What this evaluates

Evaluates Dutch LLMs on reasoning, sentiment analysis, linguistic acceptability, world knowledge, and word sense disambiguation using zero-shot multiple-choice and binary classification tasks.

Datasets

• ARC (Dutch) — total 7787; splits: test (-1)
• DBRD — total 2224; splits: test (2224)
• Dutch CoLA — total 2400; splits: test (2400); HF GroNLP/dutch-cola
• Global MMLU (Dutch) — total ?; splits: test (-1)
• XLWIC-NL — total 1004; splits: test (1004)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correct predictions out of total test instances. Computed as (number of exact matches between model output and gold label) / (total instances).
• confidence intervals — range: other
  • Statistical interval computed over 5 independent evaluation runs with sampling (temperature=1, no top_p/top_k). Reflects model confidence and prediction variance across stochastic passes.

Input / output format

Input: Zero-shot prompts in Dutch. Multiple-choice tasks present the question and predefined options. Binary tasks (sentiment, acceptability, WSD) present the query with explicit instruction to choose between two allowed labels.

Output: Constrained generation restricted to a predefined set of valid labels/options via the Outlines library. Models output exactly one allowed string (e.g., 'positive', 'negative', or the exact option text) without hallucination or need for post-processing.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    # predictions: list of 5 runs, each run is a list of model outputs
    accuracies = []
    for run_preds in predictions:
        correct = sum(1 for p, g in zip(run_preds, gold_labels) if p == g)
        accuracies.append(correct / len(gold_labels))
    mean_acc = sum(accuracies) / len(accuracies)
    ci_low = min(accuracies)
    ci_high = max(accuracies)
    return mean_acc, (ci_low, ci_high)

Common pitfalls

• Using unconstrained generation causes label hallucination; the protocol strictly requires constrained decoding (Outlines) to ensure valid outputs.
• Confidence intervals are derived from 5 stochastic runs (temperature=1), not from a single deterministic pass or bootstrap over data.
• Benchmarks like ARC and Global MMLU are machine-translated and culturally adapted, which may introduce translation artifacts not present in native Dutch evaluation.

Evidence (verbatim from paper)

Each benchmark was run five times to compute the confidence intervals. In the benchmarks, sampling was enabled (no top p or top k selection, and temperature 1). ... Outlines was used as a backbone of the benchmarking suite ... This library enables you to constrain a model’s output by regular expressions, JSON schema, or simply a list of options. ... leading to clean benchmarking results that do not require any post-processing.

Citation

@misc{vanroy2024fietje,
  title={Fietje: An open, efficient LLM for Dutch},
  author={Vanroy et al. (2024)},
  year={2024},
  note={arXiv:2412.15450}
}

• arXiv: 2412.15450