More Eval

more-eval

MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark Evaluation — He et al. (2023) (arXiv:2312.09753, 2023)

What this evaluates

Evaluates a model's ability to perform cross-modal relation extraction by predicting the semantic relationship between a textual entity in a sentence and a visual object in an image. It probes cross-modality alignment, visual-textual interaction, and handling of semantic ambiguity in multimodal fact extraction.

Datasets

• MORE — total 3559; splits: (unstated); repo https://github.com/NJUNLP/MORE

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, computed per class and averaged (macro or weighted) to handle the dataset's imbalanced label distribution. Used as the main evaluation metric.
• accuracy — range: [0, 1]
  • Proportion of correctly predicted relation tags out of total instances.
• precision — range: [0, 1]
  • Proportion of true positive predictions among all positive predictions for each relation class.
• recall — range: [0, 1]
  • Proportion of true positive predictions among all actual positive instances for each relation class.

Input / output format

Input: A sentence S containing a pre-extracted textual entity e, and an image V containing a pre-extracted visual object o.

Output: A single relation tag r from the predefined set R of 21 relation types.

Scoring recipe

def compute_metrics(preds, gold, num_classes=21):
    tp = np.sum((preds == np.arange(num_classes)) & (gold == np.arange(num_classes)), axis=0)
    fp = np.sum((preds == np.arange(num_classes)) & (gold != np.arange(num_classes)), axis=0)
    fn = np.sum((preds != np.arange(num_classes)) & (gold == np.arange(num_classes)), axis=0)
    prec = tp / (tp + fp + 1e-8)
    rec = tp / (tp + fn + 1e-8)
    f1 = 2 * prec * rec / (prec + rec + 1e-8)
    return {
        'accuracy': np.mean(preds == gold),
        'precision': np.mean(prec),
        'recall': np.mean(rec),
        'f1': np.mean(f1)
    }

Common pitfalls

• The dataset has a highly imbalanced label distribution, making accuracy misleading; F1 score is explicitly required as the primary metric.
• Models must handle pre-extracted entities and objects, meaning errors in entity/object detection or grounding can propagate to relation classification.
• Cross-modal semantic misalignment and visual ambiguity require careful attribute-aware and depth-aware encoding, which standard VLP models often lack.

Evidence (verbatim from paper)

And following the conventional MRE task, we utilize accuracy, precision, recall, and F1 value as the evaluation metrics. Since the MORE dataset has an imbalanced label distribution, we choose F1 score as the main evaluation metric for measuring the performance of a class-imbalanced task.

Citation

@misc{he2023more,
  title={MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark Evaluation},
  author={He et al. (2023)},
  year={2023},
  note={arXiv:2312.09753}
}

• arXiv: 2312.09753