Mets Cov Eval

mets-cov-eval

Named Entity Recognition in COVID-19 tweets with Entity Knowledge Augmentation — Zhang et al. (2025) (arXiv:2510.04001, 2025)

What this evaluates

Evaluates named entity recognition capabilities on biomedical and social media text. It probes the model's ability to identify and classify domain-specific entities (e.g., diseases, drugs, vaccines) in both informal tweets and formal scientific abstracts under fully-supervised and few-shot learning conditions.

Datasets

• METS-CoV — total 7000; splits: test (-1)
• BioRED — total 600; splits: test (-1)

Metrics

• Micro F1 (primary) — range: percent
  • Harmonic mean of precision and recall calculated globally across all entity types. Precision = TP / (TP + FP), Recall = TP / (TP + FN), Micro F1 = 2 * (Precision * Recall) / (Precision + Recall).

Input / output format

Input: Raw text instances (tweets or PubMed abstracts) with implicit or explicit entity boundaries to be predicted.

Output: Predicted entity spans with corresponding labels (e.g., BIO-tagged sequence or list of (start, end, type) tuples).

Scoring recipe

def compute_micro_f1(preds, golds):
    tp = fp = fn = 0
    for p, g in zip(preds, golds):
        tp += len(set(p) & set(g))
        fp += len(set(p) - set(g))
        fn += len(set(g) - set(p))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

Common pitfalls

• Micro F1 aggregates across all entity types, which can mask poor performance on rare or domain-specific classes like 'vaccine' or 'chemical entity'.
• Few-shot settings use fixed k-shot splits (5, 10, 20) without specifying random seeds, leading to potential variance in reported scores.
• The paper reports per-entity-type F1 in tables but only highlights the average/micro F1 in text, which may obscure class imbalance effects.

Evidence (verbatim from paper)

Micro F1 scores are used as our evaluation metrics. We set up 5-shot, 10-shot, and 20-shot settings for few-shot experiments.

Citation

@misc{zhang2025named,
  title={Named Entity Recognition in COVID-19 tweets with Entity Knowledge Augmentation},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.04001}
}

• arXiv: 2510.04001