Tweebank Ner Eval

tweebank-ner-eval

Annotating the Tweebank Corpus on Named Entity Recognition and Building NLP Models for Social Media Analysis — Jiang et al. (2022) (arXiv:2201.07281, 2022)

What this evaluates

Evaluates named entity recognition (NER) and syntactic NLP capabilities on noisy, informal social media text. It probes a model's ability to handle domain-specific challenges like abbreviations, irregular capitalization, and complex entity structures common in tweets, while establishing baselines for tokenization, lemmatization, POS tagging, and dependency parsing.

Datasets

• Tweebank-NER (TB2) — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/social-machines/TweebankNLP

Metrics

• entity-level F1 (primary) — range: [0, 1]
  • Standard F1 score calculated over extracted entity spans rather than individual tokens. Precision and recall are computed by comparing predicted entity sets against gold entity sets.
• F1 — range: [0, 1]
  • Token-level F1 score used for tokenization and lemmatization tasks. Computed as the harmonic mean of token-level precision and recall.
• accuracy — range: [0, 1]
  • Percentage of correctly predicted POS tags out of the total number of tokens in the test set.
• UAS — range: [0, 1]
  • Unlabeled Attachment Score: percentage of tokens where the predicted head token matches the gold head, regardless of the dependency relation label.
• LAS — range: [0, 1]
  • Labeled Attachment Score: percentage of tokens where both the predicted head token and the dependency relation label match the gold standard.

Input / output format

Input: Raw Twitter text or pre-tokenized token sequences.

Output: For NER: entity labels (PER, LOC, ORG, MISC, O) per token. For syntactic tasks: token boundaries, lemmas, UPOS tags, and dependency head/label pairs.

Scoring recipe

def entity_level_f1(preds, gold):
    pred_entities = set(extract_spans(preds))
    gold_entities = set(extract_spans(gold))
    tp = len(pred_entities & gold_entities)
    fp = len(pred_entities - gold_entities)
    fn = len(gold_entities - pred_entities)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• Using token-level F1 instead of entity-level F1 for NER evaluation, which inflates scores and misrepresents span-level performance.
• Evaluating on combined training data (TB2+WNUT17) instead of the held-out TB2 test set, violating the reported protocol.
• Ignoring domain shift: blending Twitter data with formal corpora (UD_English-EWT) often degrades performance on the TB2 test set, contrary to expectations from formal-domain baselines.

Evidence (verbatim from paper)

We pick the best models based on the corresponding dev sets and report their performance on their TB2 test sets. For each task, we compare Stanza models with existing studies and alternative NLP frameworks. Table 3: NER comparison on the TB2 test set in entity-level F1.

Citation

@misc{jiang2022tweebank,
  title={Annotating the Tweebank Corpus on Named Entity Recognition and Building NLP Models for Social Media Analysis},
  author={Jiang et al. (2022)},
  year={2022},
  note={arXiv:2201.07281}
}

• arXiv: 2201.07281