Bengal Ner El Eval

bengal-ner-el-eval

BENGAL: An Automatic Benchmark Generator for Entity Recognition and Linking — Ngoma Ngomo et al. (2017) (arXiv:1710.08691, 2017)

What this evaluates

Evaluates the performance of Named Entity Recognition (NER) and Entity Linking (EL) systems on automatically generated corpora. It probes a model's ability to accurately detect entity spans in text and correctly link them to a reference knowledge base (DBpedia) across varying document lengths, entity densities, and languages.

Datasets

• BENGAL (B1-B13, P1-P4, S1-S4) — total ?; splits: test (-1)

Metrics

• micro F1-score (primary) — range: [0, 1]
  • Micro-averaged F1 score computed globally across all entity mentions in the corpus. Precision and recall are aggregated by summing true positives, false positives, and false negatives across all documents before calculating the harmonic mean.

Input / output format

Input: Raw text documents containing natural language sentences with embedded named entities.

Output: A list of detected entity spans with their corresponding KB identifiers (e.g., DBpedia URIs) for each document.

Scoring recipe

tp = fp = fn = 0
for doc_pred, doc_gold in zip(predictions, golds):
    for span, kb_id in doc_pred:
        if (span, kb_id) in doc_gold:
            tp += 1
        else:
            fp += 1
    for span, kb_id in doc_gold:
        if (span, kb_id) not in doc_pred:
            fn += 1
prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

Common pitfalls

• The paper reports micro-averaged F1, not macro-averaged; using macro averaging will yield different results due to class imbalance across entity types.
• Evaluation is performed in an A2KB (Annotation to Knowledge Base) setting where both NER span detection and EL linking are assessed jointly; separating them changes the metric calculation.
• Datasets are dynamically generated with configurable parameters (document count, entity density, paraphrasing); results are not fixed to a single static split.

Evidence (verbatim from paper)

Then, we compared the micro F-measure of 11 NER and EL frameworks on the manually and automatically generated datasets. We chose to use these 11 frameworks because they are included in GERBIL. This inclusion ensures that their interfaces are compatible and their results comparable.

Citation

@misc{ngomangomo2017bengal,
  title={BENGAL: An Automatic Benchmark Generator for Entity Recognition and Linking},
  author={Ngoma Ngomo et al. (2017)},
  year={2017},
  note={arXiv:1710.08691}
}

• arXiv: 1710.08691