Gitm Mr Eval

gitm-mr-eval

Grounded Image Text Matching with Mismatched Relation Reasoning — Wu et al. (2023) (arXiv:2308.01236, 2023)

What this evaluates

Evaluates visual-linguistic relation understanding by requiring models to classify image-text matching, ground matched objects with bounding boxes, and identify mismatched relations from candidates. It specifically probes data efficiency and length generalization capabilities in out-of-distribution settings.

Datasets

• GITM-MR — total ?; splits: train (-1), test (-1), ood_test (-1)

Metrics

• Match% (primary) — range: percent
  • Classification accuracy for determining whether the text describes the image.
• Grounding% — range: percent
  • Recall@1 for object grounding. A case is correct only if the model predicts a match and the predicted bounding box has an IoU ≥ 0.5 with the ground-truth box.
• MRR% — range: percent
  • Top-1 accuracy for mismatch reasoning. A case is correct only if the model correctly classifies it as a mismatch and selects the exact ground-truth relation from the candidate set.

Input / output format

Input: An image and a natural language sentence describing potential objects or relations in the image.

Output: A binary classification label (match/mismatch), bounding box coordinates for matched objects, and a selected candidate relation ID for mismatched cases.

Scoring recipe

def compute_metrics(preds, gold):
    match_acc = sum(p == g for p, g in zip(preds['match'], gold['match'])) / len(gold['match'])
    ground_correct = sum(1 for p_box, g_box, p_m, g_m in zip(preds['boxes'], gold['boxes'], preds['match'], gold['match']) if p_m == g_m and iou(p_box, g_box) >= 0.5)
    grounding_acc = ground_correct / len(gold['boxes'])
    mrr_acc = sum(p == g for p, g in zip(preds['mismatch_rel'], gold['mismatch_rel'])) / len(gold['mismatch_rel'])
    return match_acc, grounding_acc, mrr_acc

Common pitfalls

• Grounding accuracy requires both correct match classification AND IoU ≥ 0.5; failing either yields zero points.
• MRR is strict top-1 accuracy, not a ranking metric; models must correctly classify mismatch AND pick the exact candidate.
• Out-of-distribution length generalization tests often reveal overfitting to training sentence lengths rather than true compositional reasoning.

Evidence (verbatim from paper)

The evaluation metrics include classification accuracy for three subtasks. The grounding result for a matched case is considered as correct when it is identified as matching and the predicted box has at least 0.5 IoU with its ground-truth location. The grounding accuracy (i.e. Recall@1) is the ratio of correctly grounded cases. For mismatch reasoning, a mismatched case needs to be correctly classified and the mismatched relation should be accurately selected from the candidate set. The MRR accuracy is the top-1 accuracy among the candidates.

Citation

@misc{wu2023grounded,
  title={Grounded Image Text Matching with Mismatched Relation Reasoning},
  author={Wu et al. (2023)},
  year={2023},
  note={arXiv:2308.01236}
}

• arXiv: 2308.01236