Norwegian Nlp Benchmark Eval

norwegian-nlp-benchmark-eval

Large-Scale Contextualised Language Modelling for Norwegian — Kutuzov et al. (2021) (arXiv:2104.06546, 2021)

What this evaluates

Evaluates contextualized language models on core Norwegian NLP tasks, probing part-of-speech tagging, named entity recognition, sentiment analysis, and negation detection across Bokmål and Nynorsk dialects.

Datasets

• Norwegian Dependency Treebank (NDT) — total ?; splits: train (-1), dev (-1), test (-1)
• NorNE — total ?; splits: train (-1), dev (-1), test (-1)
• NoReC_fine — total ?; splits: train (-1), dev (-1), test (-1)
• NoReC_sentence — total ?; splits: train (-1), dev (-1), test (-1)
• NoReC_neg — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Calculated as the number of correct token predictions divided by the total number of tokens in the test set.
• strict micro F1 (primary) — range: [0, 1]
  • Harmonic mean of micro-precision and micro-recall, strictly requiring exact match of both entity type and boundary surface string for a prediction to count as correct.
• Targeted F1 — range: [0, 1]
  • Requires correctly extracting a target span (strict) and its polarity, averaged over the test set.
• Macro F1 — range: [0, 1]
  • Harmonic mean of precision and recall calculated per class and then averaged across all classes.
• Full negation F1 — range: [0, 1]
  • Evaluates the combined prediction of negation cues and their in-sentence scopes using standard F1 scoring.

Input / output format

Input: Norwegian text sequences (sentences or documents), optionally augmented with POS tags, lemmas, or character embeddings for graph-based tasks.

Output: Task-specific predictions: token-level tags (POS), BIO sequences (NER), sentiment graph structures, or sentence-level polarity labels.

Scoring recipe

def compute_accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold)

def compute_strict_micro_f1(preds, gold):
    correct = sum(1 for p, g in zip(preds, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• ELMo models are evaluated without fine-tuning (frozen embeddings + classifier), while BERT models are fully fine-tuned, creating an unfair comparison of adaptation capacity.
• Bokmål and Nynorsk are treated as separate evaluation splits; models trained predominantly on Bokmål often show significant performance drops on Nynorsk data.

Evidence (verbatim from paper)

The evaluation metrics here is 'strict' micro F1, requiring both the correct entity type and exact match of boundary surface string.

Citation

@misc{kutuzov2021largescale,
  title={Large-Scale Contextualised Language Modelling for Norwegian},
  author={Kutuzov et al. (2021)},
  year={2021},
  note={arXiv:2104.06546}
}

• arXiv: 2104.06546