Nerel Eval

nerel-eval

NEREL: A Russian Dataset with Nested Named Entities, Relations and Events — Loukachevitch et al. (2021) (arXiv:2108.13112, 2021)

What this evaluates

Evaluates models on nested named entity recognition and relation extraction in Russian. It probes the ability to identify overlapping/contained entities and classify semantic relations between them, including cross-sentence and nested relations.

Datasets

• NEREL — total 933; splits: train (746), dev (94), test (93); repo https://github.com/nerel-ds/NEREL

Metrics

• F1 (primary) — range: percent
  • Harmonic mean of Precision and Recall: F1 = 2 * (P * R) / (P + R). Reported as a percentage.

Input / output format

Input: Full document text or single sentence, tokenized into words.

Output: For NER: set of nested entity spans with types. For RE: set of relation triples (head entity, relation type, tail entity) including nested and cross-sentence relations.

Scoring recipe

def calculate_f1(predictions, gold):
    tp = len(set(predictions) & set(gold))
    fp = len(set(predictions) - set(gold))
    fn = len(set(gold) - set(predictions))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1 * 100

Common pitfalls

• Models must handle nested/overlapping entities, which breaks standard flat NER pipelines.
• Relation extraction includes cross-sentence/document-level relations, requiring document-level context rather than just single-sentence inputs.
• The MRC approach processes one sentence at a time, which may miss document-level relations.

Evidence (verbatim from paper)

Table 3 presents the results of nested NER on the NEREL dataset. The results show, that (i) contextualized BERT-based models outperform models based on static word representations; (ii) the Biaffine model is superior to the Pyramid model; (iii) the results of MRC approach surpass nested NER models’ results, most likely, due to the effective usage of additional external information. Table 4 presents the results of relation extraction on the NEREL dataset, grouped with respect to three relation types. | Method | P | R | F1 |

Citation

@misc{loukachevitch2021nerel,
  title={NEREL: A Russian Dataset with Nested Named Entities, Relations and Events},
  author={Loukachevitch et al. (2021)},
  year={2021},
  note={arXiv:2108.13112}
}

• arXiv: 2108.13112