Multiconer2 Eval

multiconer2-eval

IXA/Cogcomp at SemEval-2023 Task 2: Context-enriched Multilingual Named Entity Recognition using Knowledge Bases — Iker García-Ferrero et al. (SemEval-2023, 2023)

What this evaluates

Evaluates a model's ability to perform fine-grained named entity recognition and entity linking across multiple languages. It probes whether external knowledge retrieval improves classification of ambiguous or low-frequency entities compared to context-only baselines.

Datasets

• MultiCoNER2 — total ?; splits: test (-1), dev (-1); repo https://github.com/ikergarcia1996/Context-enriched-NER

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score computed across all fine-grained entity categories and languages. Calculated as the unweighted mean of per-class F1 scores, where each class contributes equally regardless of frequency.

Input / output format

Input: Raw text sentences, optionally containing noise or corruption in context or entity tokens.

Output: Predicted entity spans with their corresponding fine-grained category labels (Wikidata IDs).

Scoring recipe

def compute_macro_f1(predictions, gold):
    classes = set(g['label'] for g in gold) | set(p['label'] for p in predictions)
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p['label'] == cls and g['label'] == cls)
        fp = sum(1 for p in predictions if p['label'] == cls and p not in gold)
        fn = sum(1 for g in gold if g['label'] == cls and g not in predictions)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• The evaluation distinguishes between 'clean' and 'noisy' test sets; reporting only the overall score hides robustness drops, especially in low-resource languages like Chinese.
• Entity boundary detection is evaluated separately from fine-grained classification; a high classification F1 may mask poor span extraction performance.
• The metric is computed by the shared task organizers, so local implementations may differ slightly in tokenization or span matching conventions.

Evidence (verbatim from paper)

Table 1: Our system macro-F1 score for all the tracks compared with our baseline and the systems that achieved the best results. ... Table 2: Our system macro-F1 score in the clean and noisy data from the test sets as computed by the organizers

Citation

@misc{garciaferrero2023ixa,
  title={IXA/Cogcomp at SemEval-2023 Task 2: Context-enriched Multilingual Named Entity Recognition using Knowledge Bases},
  author={Iker García-Ferrero et al.},
  year={2023},
  note={SemEval-2023}
}

• arXiv: 2304.10637