Mm Graph Eval

mm-graph-eval

Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learning — Zhu et al. (2024) (arXiv:2406.16321, 2024)

What this evaluates

Evaluates multimodal graph learning models on node classification, link prediction, and knowledge graph completion tasks. It probes the ability of models to integrate high-resolution visual and textual node features with graph structure to make accurate predictions across diverse real-world domains.

Datasets

• Amazon-Sports — total 50250; splits: train (-1), valid (-1), test (-1)
• Amazon-Cloth — total 125839; splits: train (-1), valid (-1), test (-1)
• Goodreads-LP — total 636502; splits: train (-1), valid (-1), test (-1)
• Goodreads-NC — total 685294; splits: train (-1), valid (-1), test (-1)
• Ele-fashion — total 97766; splits: train (-1), valid (-1), test (-1)
• MM-CoDEx-s — total 1383; splits: train (14298), valid (784), test (802)
• MM-CoDEx-m — total 7697; splits: train (47617), valid (2628), test (2595)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: the average of the reciprocal of the rank of the correct edge or triple among all ranked candidates.
• Hits@10 — range: [0, 1]
  • Fraction of positive edges or triples ranked within the top 10 candidates.
• Hits@1 — range: [0, 1]
  • Fraction of positive edges or triples ranked as the top candidate.
• Hits@3 — range: [0, 1]
  • Fraction of positive edges or triples ranked within the top 3 candidates.
• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted node categories out of the total number of nodes.

Input / output format

Input: Graph-structured data where each node contains textual features (e.g., product titles, book descriptions, Wikipedia text) and visual features (e.g., product images, book covers, entity images), along with adjacency information.

Output: For link prediction and KGC: a ranked list of candidate edges or triples. For node classification: a predicted category label from a fixed set (10 categories for Goodreads-NC, 12 for Ele-fashion).

Scoring recipe

def compute_mrr(ranks):
    return sum(1.0 / r for r in ranks) / len(ranks)

def compute_hits_at_k(ranks, k):
    return sum(1 for r in ranks if r <= k) / len(ranks)

def compute_accuracy(predictions, labels):
    return sum(1 for p, l in zip(predictions, labels) if p == l) / len(labels)

Common pitfalls

• Failing to explicitly remove validation and test edges from the graph during training, which causes information leakage.
• Using random negative sampling instead of hard negatives (e.g., HeaRT) for link prediction, which underestimates model difficulty and inflates scores.
• Not filtering out entities/nodes lacking multimodal features before splitting, which can introduce distribution shifts across train/val/test sets.

Evidence (verbatim from paper)

For evaluation metrics, we report MRR, Hits@10, and Hits@1, the three most commonly-used evaluation metrics for link prediction [16, 22].

Citation

@misc{zhu2024mmgraph,
  title={Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learning},
  author={Zhu et al. (2024)},
  year={2024},
  note={arXiv:2406.16321}
}

• arXiv: 2406.16321