qhjqhj00 turkish-ner-eval
Mukayese: Turkish NLP Strikes Back — Safaya et al. (2022) (arXiv:2203.01215, 2022)
What this evaluates
Tests the ability to identify and classify named entities (Person, Location, Organization) in Turkish text. It probes fine-grained token-level classification and boundary detection.
Datasets
- Milliyet-Ner — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/alisafaya/mukayese
- WikiANN (Turkish subset) — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/alisafaya/mukayese
Metrics
CoNLL F-1(primary) — range: [0, 100]- F1 score based on exact match of entity spans and labels. Counts a named entity as correct only if it is an exact match of the corresponding entity in the ground truth.
Input / output format
Input: Tokenized text with word-level annotations.
Output: Sequence of entity labels (e.g., BIO format) corresponding to each token.
Scoring recipe
def score_ner(predictions, gold):
pred_entities = extract_entities(predictions)
gold_entities = extract_entities(gold)
correct = len(set(pred_entities) & set(gold_entities))
precision = correct / len(pred_entities) if pred_entities else 0
recall = correct / len(gold_entities) if gold_entities else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return f1 * 100
Common pitfalls
- CoNLL F-1 requires exact span and label matching; partial overlaps are not counted as correct.
- Human performance is expected to be 98-99%, so models plateau quickly and small differences matter.
- Datasets use only three entity classes, limiting evaluation of fine-grained types.
Evidence (verbatim from paper)
Following previous work on Turkish NER, we report the CoNLL F-1 metric to assess our NER baselines. CoNLL F-1 counts a named entity as correct, only if it is an exact match of the corresponding entity in the ground truth.
Citation
@misc{safaya2022mukayese,
title={Mukayese: Turkish NLP Strikes Back},
author={Safaya et al. (2022)},
year={2022},
note={arXiv:2203.01215}
}
- arXiv: 2203.01215