Norne Eval

norne-eval

NorNE: Annotating Named Entities for Norwegian — Jørgensen et al. (2019) (arXiv:1911.12146, 2019)

What this evaluates

Evaluates Named Entity Recognition (NER) performance on Norwegian text, testing the model's ability to identify and classify entity boundaries and types (PER, ORG, LOC, GPE, PROD, EVT, DRV) across Bokmål and Nynorsk variants.

Datasets

• NorNE — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/ltgoslo/norne

Metrics

• F1 (strict) (primary) — range: [0, 1]
  • Entity-level F1 score requiring exact match on both predicted boundary and entity label. Follows SemEval 2013 task 9.1 strict evaluation scheme.

Input / output format

Input: Tokenized sentence with word-level and character-level representations (CNN max-pooling + pre-trained word embeddings).

Output: Sequence of BIO-style tags (IOB2, IOB, IOBE, IOBS, or IOBES) for each token, indicating entity type and position.

Scoring recipe

def strict_entity_f1(predictions, gold):
    pred_entities = extract_entities(predictions) # (start, end, label)
    gold_entities = extract_entities(gold)
    tp = sum(1 for e in pred_entities if e in gold_entities)
    fp = len(pred_entities) - tp
    fn = len(gold_entities) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• Evaluating at the token level instead of the required strict entity level.
• Interpreting higher F1 scores on coarser label sets (e.g., NorNE-6) as improved model capability, rather than a reduction in label ambiguity.
• Testing cross-standard (Bokmål to Nynorsk) performance without joint training, which artificially penalizes the model due to lexical distribution shifts.

Evidence (verbatim from paper)

For model evaluation we follow the scheme defined by the SemEval 2013 task 9.1 (Segura-Bedmar et al., 2013), using the re-implementation offered by David S. Batista. We report F1 for exact match on the entity level, i.e., both the predicted boundary and entity label must be correct. (This measure was dubbed strict in SemEval 2013 task 9.1.)

Citation

@misc{jorgensen2019norne,
  title={NorNE: Annotating Named Entities for Norwegian},
  author={Jørgensen et al. (2019)},
  year={2019},
  note={arXiv:1911.12146}
}

• arXiv: 1911.12146