Crossner Eval

crossner-eval

CrossNER: Evaluating Cross-Domain Named Entity Recognition — Liu et al. (2020) (arXiv:2012.04373, 2020)

What this evaluates

Evaluates cross-domain named entity recognition by measuring how well models adapt from a source domain (CoNLL2003) to five specialized target domains. These domains feature unique, domain-specific entity types that test the model's ability to generalize beyond standard categories.

Datasets

• CrossNER — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/zliucr/CrossNER

Metrics

• F1 score (primary) — range: [0, 1]
  • Standard exact-match F1 score for Named Entity Recognition, computed as the harmonic mean of precision and recall over all correctly identified entity spans across the target domains.

Input / output format

Input: Raw text sequences with token-level annotations for named entities.

Output: Sequence of entity type labels (e.g., BIO/IOB format) corresponding to each token in the input sequence.

Scoring recipe

def compute_f1(gold_spans, pred_spans):
    gold_set = set(gold_spans)
    pred_set = set(pred_spans)
    tp = len(gold_set & pred_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1

Common pitfalls

• Models may overfit to source domain entity types that do not exist in the target domains, causing label leakage.
• Upsampling target data in joint training can skew gradients if the source-target ratio is not carefully balanced.
• Domain-specific entity types require strict annotation guidelines to prevent cross-domain semantic drift.

Evidence (verbatim from paper)

We consider the CoNLL2003 English NER dataset... as the source domain and five domains in CrossNER as target domains. ... Then, we carry out three different settings for the domain adaptation, which are described as follows: • We ignore the source domain training samples, and fine-tune BERT directly on the target domain data. • We first pre-train BERT on the source domain data, and then fine-tune it to the target domain samples. • We jointly fine-tune BERT on both source and target domain data samples.

Citation

@misc{liu2020crossner,
  title={CrossNER: Evaluating Cross-Domain Named Entity Recognition},
  author={Liu et al. (2020)},
  year={2020},
  note={arXiv:2012.04373}
}

• arXiv: 2012.04373