Gmner Eval

gmner-eval

SAKE: Self-aware Knowledge Exploitation-Exploration for Grounded Multimodal Named Entity Recognition — Tang et al. (2026) (arXiv:2604.20146, 2026)

What this evaluates

Evaluates a model's ability to recognize named entities in multimodal social media content and ground them to visual regions, while dynamically deciding when to use internal knowledge versus external search tools.

Datasets

• Twitter-GMNER* — total ?; splits: test (-1)
• Twitter-FMNERG* — total ?; splits: test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for predicted entity spans, types, and grounding references. Calculated separately for Multimodal Named Entity Recognition (MNER) and Entity Extraction and Grounding (EEG) subtasks, then aggregated for overall GMNER performance.

Input / output format

Input: Multimodal inputs consisting of social media images and associated text, with ground-truth entity annotations for evaluation.

Output: Predicted entity spans, entity types, and grounding references (e.g., bounding boxes or image region IDs) for each instance.

Scoring recipe

def compute_f1(predictions, gold):
    pred_spans = set(predictions)
    gold_spans = set(gold)
    tp = len(pred_spans & gold_spans)
    precision = tp / len(pred_spans) if pred_spans else 0
    recall = tp / len(gold_spans) if gold_spans else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• Models may over-rely on external search tools without self-awareness of knowledge gaps, leading to high search costs and noise.
• The search penalty hyperparameter significantly trades off F1 performance against search ratio; improper tuning yields either excessive retrieval or degraded accuracy.
• Difficulty-aware search tag generation requires careful calibration of sampling level N to balance cold-start data distribution and RL training stability.

Evidence (verbatim from paper)

We evaluate our approach on two benchmarks, Twitter-GMNER*(gmner)* and Twitter-FMNERG*(fg-gmner). The details of datasets are provided in Appendix[D.1]. We follow the standard evaluation protocol for GMNER as described in(gmner)*, assessing performance on both Multimodal Named Entity Recognition (MNER) and Entity Extraction and Grounding (EEG). Overall performance is reported using F1 scores.

Citation

@misc{tang2026sake,
  title={SAKE: Self-aware Knowledge Exploitation-Exploration for Grounded Multimodal Named Entity Recognition},
  author={Tang et al. (2026)},
  year={2026},
  note={arXiv:2604.20146}
}

• arXiv: 2604.20146