muiE-eval

muiE-eval

Recognizing Everything from All Modalities at Once: Grounded Multimodal Universal Information Extraction — Zhang et al. (2024) (arXiv:2406.03701, 2024)

What this evaluates

This benchmark evaluates a model's ability to perform universal information extraction (NER, RE, EE) and fine-grained cross-modal grounding (segmentation/tracking) across text, image, audio, and video modalities in a unified zero-shot setting. It probes the model's capacity to align semantic information with visual/auditory content and handle modality-shared versus modality-specific scenarios without task-specific fine-tuning.

Datasets

• MUIE — total 3000; splits: test (3000)

Metrics

• F1 (NER) (primary) — range: [0, 1]
  • F1 score computed over exact matches of entity spans and their corresponding type labels. Precision and recall are calculated based on correctly predicted (span, type) pairs.
• F1 (RE) — range: [0, 1]
  • F1 score over exact matches of subject entities, object entities, and their relation labels. All three components must match the gold triple.
• F1 (EE) — range: [0, 1]
  • Two separate F1 scores: Event Trigger (ET) F1 requires matching both trigger span and event type; Event Argument (EA) F1 requires matching argument spans and their role types.
• mIoU — range: [0, 1]
  • Mean Intersection over Union for image and audio segmentation. Computed as the average of IoU scores across all predicted and ground-truth masks.
• Jaccard — range: [0, 1]
  • Average Jaccard index for video segmentation/tracking, calculated identically to IoU over frame-level or track-level binary masks.

Input / output format

Input: Multimodal inputs consisting of text, image, audio, video, or combinations (e.g., T+I, T+A, T+V, I+A, V+A), accompanied by UIE instruction prompts specifying the extraction task and candidate label sets.

Output: Structured text meta-response containing extracted entities/relations/events in formats like (entity, label) or (subject, Relation, object), followed by grounding coordinates/masks for the corresponding modalities.

Scoring recipe

def compute_f1(preds, golds):
    correct = len(set(preds) & set(golds))
    prec = correct / len(preds) if preds else 0
    rec = correct / len(golds) if golds else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_iou(pred_mask, gold_mask):
    inter = len(set(pred_mask) & set(gold_mask))
    union = len(set(pred_mask) | set(gold_mask))
    return inter / union if union > 0 else 0

Common pitfalls

• Error propagation: Pipeline systems first generate a meta-response and then invoke grounding modules, causing errors in extraction to cascade into incorrect grounding.
• Modality misalignment: Models often struggle when information across modalities is shared versus specific, leading to incorrect grounding matches or incomplete extraction.
• Zero-shot constraint: All baselines and the proposed model are evaluated without tuning on in-house datasets, making performance highly sensitive to instruction-following rather than task-specific fine-tuning.

Evidence (verbatim from paper)

We measure the system performance by following most practices of end-to-end UIE: F1 of entity span with type for NER, F1 of all subject&object entities and their relation label for RE. For EE, we consider event trigger (ET) F1 including both trigger and event type, and event argument F1 including both arguments with role types. W.r.t. the multimodal grounding, for both image and audio segmentation, we consider the mean Intersection over Union (mIoU); for video segmentation, we use the average Jaccard (J).

Citation

@misc{zhang2024muiE,
  title={Recognizing Everything from All Modalities at Once: Grounded Multimodal Universal Information Extraction},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2406.03701}
}

• arXiv: 2406.03701