qhjqhj00 finnish-news-ner-eval
A Finnish News Corpus for Named Entity Recognition — Ruokolainen et al. (2019) (arXiv:1908.04212, 2019)
What this evaluates
Evaluates named entity recognition systems on Finnish text, testing their ability to identify and classify entities (person, location, organization, product, event, date) in both in-domain news and out-of-domain Wikipedia corpora. It specifically probes domain generalization and the handling of nested entity spans.
Datasets
- Finnish News Corpus — total 193742; splits: train (183552), dev (10190), test_digitoday (46363), test_wikipedia (49752)
Metrics
F1-score(primary) — range: percent- Harmonic mean of precision and recall. Precision = correctly recognized entities / all recognized entities; Recall = correctly recognized entities / all annotated entities. Overall F1 is computed via micro averaging across all entity classes.
Input / output format
Input: Preprocessed plain text articles.
Output: Span-level named entity annotations with labels: PER, LOC, ORG, PRO, EVENT, DATE. Supports nested entities.
Scoring recipe
def compute_f1(pred_spans, gold_spans):
tp = len([s for s in pred_spans if s in gold_spans])
fp = len([s for s in pred_spans if s not in gold_spans])
fn = len([s for s in gold_spans if s not in pred_spans])
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return f1
# Overall F1 uses micro-averaging: sum TP, FP, FN across all classes before computing P, R, F1.
Common pitfalls
- Models may ignore nested entities during training/prediction, yielding different scores than systems that predict them.
- Overall F1 can be computed via macro or micro averaging; the protocol explicitly follows CoNLL-2003 using micro averaging.
- Evaluating only top-level entities versus all entities (including nested) yields significantly different F1 scores.
Evidence (verbatim from paper)
We follow the classic CoNLL-2003 shared task (Tjong Kim Sang and De Meulder, 2003) and evaluate the systems using F1-score which is the harmonic mean of precision (the number of correctly recognized entities divided by the number of all recognized entities) and recall (the number of correctly recognized entities divided by the number of all annotated entities in data). We compute the precision, recall, and F1 measures for each entity class (PER, LOC, ORG, PRO, EVENT, DATE) individually and over all classes to assess the overall performance of each system. Moreover, the measures over all classes can, in general, be obtained in two different ways using either macro or micro averaging... The CoNLL-2003 evaluation employs the micro averaging approach. The evaluations are performed on all entities as well as on top-level entities only.
Citation
@misc{ruokolainen2019finnish,
title={A Finnish News Corpus for Named Entity Recognition},
author={Ruokolainen et al. (2019)},
year={2019},
note={arXiv:1908.04212}
}
- arXiv: 1908.04212