Stackoverflow Ner Eval

stackoverflow-ner-eval

Code and Named Entity Recognition in StackOverflow — Tabassum et al. (2020) (arXiv:2005.01634, 2020)

What this evaluates

Evaluates named entity recognition capabilities on software programming texts. It specifically probes the model's ability to identify fine-grained code-related entities like variable names, libraries, and data structures in StackOverflow posts.

Datasets

• StackOverflow NER corpus — total 15409; splits: train (9352), dev (2942), test (3115); repo https://github.com/jeniyat/StackOverflowNER

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall at the token level. Precision is the proportion of predicted entities that are correct; recall is the proportion of gold entities that are correctly identified.
• Precision — range: [0, 1]
  • Proportion of predicted entity tokens that match the gold standard.
• Recall — range: [0, 1]
  • Proportion of gold entity tokens that are correctly predicted.

Input / output format

Input: A sentence or token from a StackOverflow post, optionally with surrounding context.

Output: Sequence of BIO-style entity tags corresponding to each token in the input sentence.

Scoring recipe

def compute_ner_f1(predictions, gold):
    correct = 0
    pred_count = 0
    gold_count = 0
    for pred_sent, gold_sent in zip(predictions, gold):
        pred_entities = extract_entities(pred_sent)
        gold_entities = extract_entities(gold_sent)
        correct += len(set(pred_entities) & set(gold_entities))
        pred_count += len(pred_entities)
        gold_count += len(gold_entities)
    precision = correct / pred_count if pred_count > 0 else 0
    recall = correct / gold_count if gold_count > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return {'precision': precision, 'recall': recall, 'f1': f1}

Common pitfalls

• Domain shift: models trained on newswire (e.g., CoNLL 2003) perform poorly on code text due to polysemous tokens and unseen code identifiers.
• Segmentation errors: models often mispredict entity boundaries in code-heavy sentences, which heavily impacts F1.
• Unseen tokens: 38% of tokens inside gold entity spans are unseen during training, requiring robust contextual embeddings.

Evidence (verbatim from paper)

We train and evaluate our SoftNER model on the StackOverflow NER corpus of 9,352 train, 2,942 development and 3,115 test sentences we constructed in §2. Table 2 shows the precision (P), recall (R) and F1 score comparison of different models evaluated on the StackOverflow NER corpus.

Citation

@misc{tabassum2020stackoverflowner,
  title={Code and Named Entity Recognition in StackOverflow},
  author={Tabassum et al. (2020)},
  year={2020},
  note={arXiv:2005.01634}
}

• arXiv: 2005.01634