# Finnish Nlp Eval

> finnish-nlp-eval

- Skill: `qhjqhj00/finnish-nlp-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/finnish-nlp-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/finnish-nlp-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/finnish-nlp-eval

---


# finnish-nlp-eval

> Towards Fully Bilingual Deep Language Modeling — Chang et al. (2020) (arXiv:2010.11639, 2020)

## What this evaluates

Evaluates Finnish NLP capabilities across four core tasks: part-of-speech tagging, named entity recognition, dependency parsing, and text classification on domain-specific and out-of-domain corpora. It probes a model's ability to handle morphologically rich language and varying text registers.

## Datasets

- **Finnish NLP Benchmarks** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `UPOS` — range: [0, 1]
  - Unambiguous Part-Of-Speech tag accuracy for POS tagging on Universal Dependencies treebanks.
- `F-score` — range: [0, 1]
  - Mention-level F-score for Named Entity Recognition using IOB annotations.
- `LAS` — range: [0, 1]
  - Labeled Attachment Score for dependency parsing (accuracy of predicted heads and relations).
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified documents for text classification.
- `Average` **(primary)** — range: [0, 1]
  - Algebraic mean of the four task scores (POS, NER, parsing, classification).

## Input / output format

**Input**: Tokenized Finnish text, typically truncated to at most 256 tokens.

**Output**: Task-specific predictions: POS tags, IOB entity tags, dependency heads/relations, or document class labels.

## Scoring recipe

```python
pos_score = upos_accuracy(y_true, y_pred)
ner_score = mention_level_f1(y_true, y_pred)
parse_score = labeled_attachment_score(y_true, y_pred)
cls_score = document_accuracy(y_true, y_pred)
average = (pos_score + ner_score + parse_score + cls_score) / 4
```

## Common pitfalls

- Uses gold segmentation for dependency parsing instead of predicted segmentation.
- Truncates all documents to 256 tokens, which may disadvantage models with longer context windows.
- Averages different metrics (UPOS, F1, LAS, Accuracy) without normalization, which can skew the final score.

## Evidence (verbatim from paper)

> The metric for POS tagging is UPOS, NER F-score, dependency parsing LAS, and text classification accuracy.

## Citation

```bibtex
@misc{chang2020bilingual,
  title={Towards Fully Bilingual Deep Language Modeling},
  author={Chang et al. (2020)},
  year={2020},
  note={arXiv:2010.11639}
}
```

- arXiv: 2010.11639