Zs Multimodal Ie Eval

zs-multimodal-ie-eval

Multimodal Graph-Based Variational Mixture of Experts Network for Zero-Shot Multimodal Information Extraction — Baohang Zhou et al. (2025) (arXiv:2502.15290, 2025)

What this evaluates

Evaluates zero-shot multimodal named entity typing and relation extraction. It probes a model's ability to align text and image modalities for fine-grained semantic recognition of unseen entity types and relations without additional training.

Datasets

• WikiDiverse — total ?; splits: train (-1), val (-1), test (-1)
• Zheng et al. MRE dataset — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• F1 (primary) — range: percent
  • Macro-averaged F1 score across all categories. Computed as the harmonic mean of macro-averaged precision and recall: F1 = 2 * (P * R) / (P + R).
• Precision — range: percent
  • Macro-averaged precision across categories, representing the proportion of correct predictions among all positive predictions.
• Recall — range: percent
  • Macro-averaged recall across categories, representing the proportion of actual positives correctly identified.
• Accuracy — range: percent
  • Macro-averaged accuracy across categories, representing the proportion of correctly classified instances.

Input / output format

Input: Text-image pair containing an annotated entity mention (for MET) or two entity mentions with surrounding context (for MRE).

Output: Predicted fine-grained entity type (for MET) or relation type (for MRE) from the zero-shot category set.

Scoring recipe

def compute_macro_metrics(y_true, y_pred, classes):
    p_list, r_list, f1_list, acc_list = [], [], [], []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        p = tp / (tp + fp) if (tp + fp) > 0 else 0
        r = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
        acc = tp / (tp + fp + fn) if (tp + fp + fn) > 0 else 0
        p_list.append(p); r_list.append(r); f1_list.append(f1); acc_list.append(acc)
    return np.mean(p_list), np.mean(r_list), np.mean(f1_list), np.mean(acc_list)

Common pitfalls

• The zero-shot split is performed at the category level rather than the instance level, so random seed choice significantly impacts results (hence 3 runs are reported).
• Both datasets originally contain 'Other' or 'None' categories that are explicitly excluded, altering the effective label space and requiring careful filtering before evaluation.
• Metrics are macro-averaged, which can mask performance on minority categories if the category distribution is imbalanced.

Evidence (verbatim from paper)

For these tasks, we undertake experiments utilizing the respective benchmark datasets. For MET task, we utilize the WikiDiverse [21] as the benchmark dataset... To compare our model with the baselines under the zero-shot setting, we mimic this scenario by randomly splitting original category set into three parts... We evaluated MG-VMoE alongside baseline models on the MET and MRE benchmark datasets, and reported macro-averaged precision (P), recall (R), F1 scores, and accuracy, considering the varying sample sizes across different categories.

Citation

@misc{zhou2025mgvmoe,
  title={Multimodal Graph-Based Variational Mixture of Experts Network for Zero-Shot Multimodal Information Extraction},
  author={Baohang Zhou et al. (2025)},
  year={2025},
  note={arXiv:2502.15290}
}

• arXiv: 2502.15290