Scanner Mner Eval

scanner-mner-eval

SCANNER: Knowledge-Enhanced Approach for Robust Multi-modal Named Entity Recognition of Unseen Entities — Ok et al. (2024) (arXiv:2404.01914, 2024)

What this evaluates

Evaluates multi-modal named entity recognition (MNER) and visual grounding capabilities, specifically probing the model's ability to generalize to unseen entities by leveraging external knowledge (Wikipedia) and image-based features.

Datasets

• MNER — total ?; splits: train (-1), test (-1)
• GMNER — total ?; splits: train (-1), test (-1)

Metrics

• F1 score (primary) — range: percent
  • Harmonic mean of precision and recall (2 * (Precision * Recall) / (Precision + Recall)).

Input / output format

Input: Text prompt formatted as: 'The entity is [mask] for {entity} in this sentence. {original sentence}{Wikipedia}{image caption} [obj] {object 1} [obj] {object 2} …', where knowledge sources are concatenated.

Output: Entity class distribution (BIO tagging for stage 1, class probabilities for stage 2) and overlap scores (predicted IoU) for visual grounding.

Scoring recipe

def compute_f1(precision, recall):
    if precision + recall == 0:
        return 0.0
    return 2 * (precision * recall) / (precision + recall)

# For each dataset split:
# 1. Compute precision and recall from predicted vs. gold entity spans/classes.
# 2. Apply compute_f1 to get the headline metric.
# 3. Report as percentage.

Common pitfalls

• Noisy annotations in datasets can mislead training, especially at ambiguous entity boundaries; the paper addresses this via the Trust Your Teacher distillation method.
• Visual grounding requires matching entity candidates to specific image regions, which can be ambiguous if irrelevant objects are referenced.

Evidence (verbatim from paper)

| Methods | Twitter-2015 | | | Twitter-2017 | | | Twitter-GMNER | | | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | | Pre. | Rec. | F1 | Pre. | Rec. | F1 | Pre. | Rec. | F1 |

Citation

@misc{ok2024scanner,
  title={SCANNER: Knowledge-Enhanced Approach for Robust Multi-modal Named Entity Recognition of Unseen Entities},
  author={Ok et al. (2024)},
  year={2024},
  note={arXiv:2404.01914}
}

• arXiv: 2404.01914