Continual Ner Eval

continual-ner-eval

Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition — Zheng et al. (2022) (arXiv:2210.03980, 2022)

What this evaluates

Evaluates a model's ability to perform continual learning in Named Entity Recognition (CL-NER) by incrementally learning new entity types while mitigating catastrophic forgetting of previously learned types. It specifically probes how well the model handles the 'Other-class' (miscellaneous/old entities) during incremental training and maintains performance across sequential learning steps.

Datasets

• OntoNotes5 — total ?; splits: train (-1), val (-1), test (-1)
• i2b2 — total ?; splits: train (-1), val (-1), test (-1)
• CoNLL2003 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Micro F1 (primary) — range: percent
  • Standard token-level F1 score computed over all predicted and gold entity labels, averaged across all continual learning steps (including the first step).
• Macro F1 — range: percent
  • Standard token-level F1 score computed per entity type and then averaged across all types, averaged across all continual learning steps.

Input / output format

Input: Tokenized text sequences with corresponding token-level entity labels. During training, data is partitioned into disjoint slices where only new entity types are retained and old types are relabeled as Other-class.

Output: Per-token entity label predictions (e.g., B-PER, I-LOC, O/Other-class).

Scoring recipe

def compute_f1(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == g and p != 'O')
    fp = sum(1 for p, g in zip(preds, golds) if p != g and p != 'O')
    fn = sum(1 for p, g in zip(preds, golds) if p != g and g != 'O')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) * 100

# Average across all CL steps as specified
step_scores = [compute_f1(step_preds, step_golds) for step_preds, step_golds in all_steps]
final_metric = sum(step_scores) / len(step_scores)

Common pitfalls

• Failing to average the F1 scores across all continual learning steps (including the initial step) as explicitly required by the protocol.
• Incorrectly handling the 'Other-class' during evaluation: the paper specifies that during testing, all recognized entity types are retained, whereas during training slices, old types are relabeled as Other-class.
• Not filtering out entity types with fewer than 50 training samples before partitioning, which alters the dataset composition and slice distribution.

Evidence (verbatim from paper)

Considering the class imbalance problem in NER, we adopt Micro F1 and Macro F1 for measuring the model performance. We report the average result on all CL steps (including the first step) as the final result.

Citation

@misc{zheng2022distilling,
  title={Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition},
  author={Zheng et al. (2022)},
  year={2022},
  note={arXiv:2210.03980}
}

• arXiv: 2210.03980