Ner Trigger Efficiency Eval

ner-trigger-efficiency-eval

TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition — Lin et al. (2020) (arXiv:2004.07493, 2020)

What this evaluates

Evaluates the data-efficiency and labor-cost effectiveness of a trigger-enhanced Named Entity Recognition model compared to a standard baseline. It probes how well the model generalizes when trained on varying fractions of labeled sentences and trigger-annotated data.

Datasets

• CoNLL2003 — total ?; splits: train (-1)
• BC5CDR — total ?; splits: train (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall over all correctly identified entity mentions. F1 = 2 * (Precision * Recall) / (Precision + Recall).

Input / output format

Input: Tokenized sentence sequences.

Output: Sequence of entity tags (e.g., BIO format) for each token in the input sentence.

Scoring recipe

def compute_f1(pred_tags, gold_tags):
    pred_ents = extract_mentions(pred_tags)
    gold_ents = extract_mentions(gold_tags)
    tp = len(pred_ents & gold_ents)
    fp = len(pred_ents - gold_ents)
    fn = len(gold_ents - pred_ents)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

Common pitfalls

• Results are reported as curves across varying training data fractions (5% to 70%), not as a single fixed score.
• Trigger annotation requires approximately 1.5x human effort compared to standard entity tagging, which must be factored into labor-efficiency comparisons.
• Self-training uses a fixed top-20% confidence threshold per epoch, which may not generalize without dataset-specific tuning.

Evidence (verbatim from paper)

Even if we consider the extreme case that tagging triggers requires twice the human effort ("BLSTM-CRF (x2)"), the TMN is still significantly more labor-efficient in terms of F1 scores.

Citation

@misc{lin2020triggerner,
  title={TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition},
  author={Lin et al. (2020)},
  year={2020},
  note={arXiv:2004.07493}
}

• arXiv: 2004.07493