E2e Gmner Eval

e2e-gmner-eval

E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition — Meng Zhang et al. (2026) (arXiv:2604.17319, 2026)

What this evaluates

Evaluates a model's ability to perform end-to-end grounded multimodal named entity recognition, jointly identifying entity spans in text, predicting their semantic types, and grounding them to corresponding bounding boxes in an associated image. It probes the model's capacity for multimodal alignment, structured generation, and robustness to annotation noise via chain-of-thought reasoning.

Datasets

• Twitter-GMNER — total ?; splits: (unstated); repo https://github.com/Finch-coder/E2E-GMNER
• Twitter-FMNERG — total ?; splits: (unstated); repo https://github.com/Finch-coder/E2E-GMNER

Metrics

• GMNER (primary) — range: percent
  • Joint F1 score for the complete GMNER task, requiring exact or IoU-matched matching of entity spans, semantic types, and bounding boxes.
• MNER — range: percent
  • Multimodal NER F1 score, evaluating only the correctness of entity span detection and semantic type prediction, ignoring grounding.
• EEG — range: percent
  • Entity Entity Grounding F1 score, evaluating the accuracy of bounding box predictions for correctly identified entities, typically using an IoU threshold.

Input / output format

Input: An image-text pair (I, T) accompanied by a task-specific instruction.

Output: A single autoregressive sequence starting with chain-of-thought reasoning R, followed by concatenated structured entity records formatted as: e_i | c_i | [x_i^1, y_i^1, x_i^2, y_i^2].

Scoring recipe

def score(predictions, gold):
    pred_entities = parse_output(predictions)
    gold_entities = parse_output(gold)
    matched_pred, matched_gold = match_entities(pred_entities, gold_entities, type_match=True)
    iou_threshold = 0.5
    tp = sum(1 for p, g in zip(matched_pred, matched_gold) if iou(p.box, g.box) >= iou_threshold)
    precision = tp / len(matched_pred) if matched_pred else 0
    recall = tp / len(matched_gold) if matched_gold else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1 * 100

Common pitfalls

• The model generates an unordered set of entities, so evaluation must handle permutation invariance when matching predictions to gold.
• Bounding boxes are predicted as discrete tokens; exact coordinate matching is too strict due to discretization, so IoU-based matching is required.
• Chain-of-thought reasoning is generated during inference but is not part of the structured output schema, so it must be stripped before parsing entity records.

Evidence (verbatim from paper)

Grounded Multimodal Named Entity Recognition (GMNER) aims to jointly identify named entity mentions in text, predict their semantic types, and ground each entity to a corresponding visual region in the associated image.

Citation

@misc{zhang2026e2egmner,
  title={E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition},
  author={Meng Zhang et al. (2026)},
  year={2026},
  note={arXiv:2604.17319}
}

• arXiv: 2604.17319