Scandeval Benchmark Eval

scandeval-benchmark-eval

ScandEval: A Benchmark for Scandinavian Natural Language Processing — Nielsen et al. (2023) (arXiv:2304.00906, 2023)

What this evaluates

Evaluates the performance of monolingual and multilingual language models across five Scandinavian languages (Danish, Norwegian, Swedish, Icelandic, Faroese) on question answering, linguistic acceptability, and named entity recognition. It also probes cross-lingual transfer capabilities between these languages by measuring performance variance across language groups.

Datasets

• ScandiQA — total ?; splits: test (-1)
• ScaLA — total ?; splits: test (-1)
• MIM-GOLD-NER — total ?; splits: test (-1)
• WikiANN — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy or F1 score per task (exact computation detailed in Section 3 of the paper).
• F-statistic — range: other
  • ANOVA F-statistic computed on per-language model scores to quantify variance in performance across language groups, used to assess cross-lingual transfer.

Input / output format

Input: Task-specific inputs: for QA, a question and context; for acceptability, a sentence; for NER, a sentence. All inputs are in one of the five Scandinavian languages.

Output: Task-specific outputs: predicted answer/label or entity spans. Scores are aggregated per language and overall.

Scoring recipe

def compute_scores(predictions, gold, task):
    if task in ['qa', 'acceptability']:
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task == 'ner':
        return compute_f1(predictions, gold)  # Exact method in Sec 3
# Aggregate per language and overall
# Cross-lingual transfer: compute F-statistic across language groups

Common pitfalls

• The paper reports F-statistics for cross-lingual transfer, which measure variance between language groups rather than direct model accuracy.
• Confidence intervals are reported alongside scores but are not explicitly defined in the provided section.
• Evaluation covers both Mainland (da, no, sv) and Insular (is, fo) languages, which show markedly different transfer capabilities and should not be averaged without stratification.

Evidence (verbatim from paper)

These scores have been computed as described in Section 3, and the top-5 performing models for each language, as well as overall, can be found in Table 1. ... The resulting benchmark results can be found in Table 2 and all the raw scores can be found in the appendix. The results affirm our two hypotheses, as we see that the group of languages with the largest F-statistic is the group of Mainland Scandinavian languages.

Citation

@misc{nielsen2023scandeval,
  title={ScandEval: A Benchmark for Scandinavian Natural Language Processing},
  author={Nielsen et al. (2023)},
  year={2023},
  note={arXiv:2304.00906}
}

• arXiv: 2304.00906