Finnish Nlp Eval

finnish-nlp-eval

Towards Fully Bilingual Deep Language Modeling — Chang et al. (2020) (arXiv:2010.11639, 2020)

What this evaluates

Evaluates Finnish NLP capabilities across four core tasks: part-of-speech tagging, named entity recognition, dependency parsing, and text classification on domain-specific and out-of-domain corpora. It probes a model's ability to handle morphologically rich language and varying text registers.

Datasets

• Finnish NLP Benchmarks — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• UPOS — range: [0, 1]
  • Unambiguous Part-Of-Speech tag accuracy for POS tagging on Universal Dependencies treebanks.
• F-score — range: [0, 1]
  • Mention-level F-score for Named Entity Recognition using IOB annotations.
• LAS — range: [0, 1]
  • Labeled Attachment Score for dependency parsing (accuracy of predicted heads and relations).
• Accuracy — range: [0, 1]
  • Proportion of correctly classified documents for text classification.
• Average (primary) — range: [0, 1]
  • Algebraic mean of the four task scores (POS, NER, parsing, classification).

Input / output format

Input: Tokenized Finnish text, typically truncated to at most 256 tokens.

Output: Task-specific predictions: POS tags, IOB entity tags, dependency heads/relations, or document class labels.

Scoring recipe

pos_score = upos_accuracy(y_true, y_pred)
ner_score = mention_level_f1(y_true, y_pred)
parse_score = labeled_attachment_score(y_true, y_pred)
cls_score = document_accuracy(y_true, y_pred)
average = (pos_score + ner_score + parse_score + cls_score) / 4

Common pitfalls

• Uses gold segmentation for dependency parsing instead of predicted segmentation.
• Truncates all documents to 256 tokens, which may disadvantage models with longer context windows.
• Averages different metrics (UPOS, F1, LAS, Accuracy) without normalization, which can skew the final score.

Evidence (verbatim from paper)

The metric for POS tagging is UPOS, NER F-score, dependency parsing LAS, and text classification accuracy.

Citation

@misc{chang2020bilingual,
  title={Towards Fully Bilingual Deep Language Modeling},
  author={Chang et al. (2020)},
  year={2020},
  note={arXiv:2010.11639}
}

• arXiv: 2010.11639