Norwegian Transformer Eval

norwegian-transformer-eval

Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model — Kummervold et al. (2021) (arXiv:2104.09617, 2021)

What this evaluates

Evaluates the cross-lingual transfer and domain adaptation capabilities of a Norwegian BERT model against multilingual and monolingual baselines on token-level (NER/POS) and sequence-level (sentiment/political affiliation) classification tasks.

Datasets

• NorNE — total ?; splits: test (-1)
• CoNLL-2003 — total ?; splits: test (-1)
• NoReC — total ?; splits: test (-1)
• Norwegian Parliament Speeches — total 6000; splits: test (6000)

Metrics

• F1 score (primary) — range: [0, 1]
  • Standard token-level or entity-level F1 score computed between predicted and gold entity/span labels. Macro-averaged across classes.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified text sequences out of the total number of sequences in the evaluation set.

Input / output format

Input: Tokenized text sequences for token classification; raw text sentences or paragraphs for sequence classification.

Output: Token-level entity labels (e.g., B-PER, I-ORG) for NER; discrete class labels (e.g., sentiment polarity or political party affiliation) for sequence classification.

Scoring recipe

def compute_f1(gold, pred):
    tp = sum(1 for g, p in zip(gold, pred) if g == p and g != 'O')
    fp = sum(1 for g, p in zip(gold, pred) if g != p and p != 'O')
    fn = sum(1 for g, p in zip(gold, pred) if g != p and g != 'O')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

def compute_accuracy(gold, pred):
    return sum(1 for g, p in zip(gold, pred) if g == p) / len(gold)

Common pitfalls

• Varying tag sets and annotation guidelines across languages make direct cross-lingual performance comparisons unreliable.
• Fine-tuning hyperparameters (learning rate, epochs) are highly dataset-dependent and require careful validation-set tuning.
• OCR noise in historical corpora can significantly degrade model performance if confidence-based filtering is not applied during pre-training.

Evidence (verbatim from paper)

The optimal learning rate and number of epochs mainly depend on the size of and variance in the training corpus... we could still compare the performance of our model against that of English-only and multilingual BERT models.

Citation

@misc{kummervold2021operationalizing,
  title={Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model},
  author={Kummervold et al. (2021)},
  year={2021},
  note={arXiv:2104.09617}
}

• arXiv: 2104.09617