Histnero Eval

histnero-eval

HistNERo: Historical Named Entity Recognition for the Romanian Language — Avram et al. (2024) (arXiv:2405.00155, 2024)

What this evaluates

Evaluates the ability of language models to recognize and classify named entities (PERSON, ORGANIZATION, LOCATION, PRODUCT, DATE) in historical Romanian newspaper texts across four distinct geographical regions.

Datasets

• HistNERo — total ?; splits: test (-1)

Metrics

• strict F1-score (primary) — range: percent
  • Computes the harmonic mean of precision and recall for exact matches of named entity boundaries and types. Strict F1 requires both the span boundaries and the entity type to match exactly between prediction and gold.
• accuracy — range: percent
  • Token-level accuracy measuring the proportion of correctly labeled tokens out of the total number of tokens.

Input / output format

Input: Raw text from historical Romanian newspapers, tokenized for NER processing.

Output: Token-level sequence labels for five entity types: PERSON, ORGANIZATION, LOCATION, PRODUCT, DATE.

Scoring recipe

def strict_f1(preds, golds):
    pred_ents = set(extract_entities(preds))
    gold_ents = set(extract_entities(golds))
    tp = len(pred_ents & gold_ents)
    fp = len(pred_ents - gold_ents)
    fn = len(gold_ents - pred_ents)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return f1 * 100

Common pitfalls

• Strict F1 requires exact boundary and type matching; partial overlaps or type mismatches are counted as errors.
• Performance varies significantly across regions (e.g., Transylvania vs. Moldavia), so reporting only the aggregate score masks regional linguistic disparities.
• Domain adaptation techniques (gradient/loss reversal) are applied during training, not just evaluation, which affects the reported metrics.

Evidence (verbatim from paper)

We measure the accuracy and the strict F1-score when we evaluate based on the regions and only the strict F1-score when we evaluate based on the named entities.

Citation

@misc{avram2024histnero,
  title={HistNERo: Historical Named Entity Recognition for the Romanian Language},
  author={Avram et al. (2024)},
  year={2024},
  note={arXiv:2405.00155}
}

• arXiv: 2405.00155