Multimodal Grounding Eval

multimodal-grounding-eval

Kosmos-2: Grounding Multimodal Large Language Models to the World — Peng et al. (2023) (arXiv:2306.14824, 2023)

What this evaluates

Evaluates a model's ability to localize text phrases and referring expressions within images by generating bounding box coordinates, and conversely to generate text descriptions from given bounding boxes. It probes spatial reasoning, text-image alignment, and zero-shot generalization across different expression types.

Datasets

• Flickr30k Entities — total ?; splits: val (-1), test (-1)
• RefCOCO — total ?; splits: val (-1), testA (-1), testB (-1)
• RefCOCO+ — total ?; splits: val (-1), testA (-1), testB (-1)
• RefCOCOg — total ?; splits: val (-1), test (-1)

Metrics

• R@1 (primary) — range: percent
  • Recall at top-1 generated bounding box. A prediction is counted as correct if its Intersection over Union (IoU) with the ground-truth box exceeds 0.5.
• R@5 — range: percent
  • Recall at top-5 generated bounding boxes. If fewer than 5 boxes are generated, all available boxes are used for calculation.
• R@10 — range: percent
  • Recall at top-10 generated bounding boxes. Uses all available boxes if fewer than 10 are generated.
• Accuracy — range: percent
  • For referring expression comprehension, accuracy is measured using only the first generated bounding box, correct if IoU > 0.5.

Input / output format

Input: Image embedding followed by a grounding prompt. For phrase grounding: ' Image Embedding ...{phrase}' where preceding words provide context. For referring expression comprehension: '{referring expression}'.

Output: Model generates location tokens enclosed in '...' tags, which are parsed into bounding boxes. For generation tasks, it outputs natural language text descriptions.

Scoring recipe

def compute_recall(predictions, gold_boxes):
    correct = 0
    for pred_box in predictions[:10]:
        if pred_box is None:
            break
        iou = compute_iou(pred_box, gold_boxes)
        if iou > 0.5:
            correct += 1
            break
    return (correct / len(gold_boxes)) * 100

Common pitfalls

• Using isolated phrases without preceding context increases ambiguity; the protocol explicitly requires including preceding words as context.
• Malformed or unconvertible location sequences (e.g., '') are treated as negative samples and must be filtered out before scoring.
• RefCOCO/RefCOCO+ datasets use shorter expressions from a two-player game, which inherently leads to lower performance compared to RefCOCOg.

Evidence (verbatim from paper)

We obtain the location tokens in “..” from the model response and then covert it into bounding boxes. The generated bounding box is correct if its intersection over union (IoU) with the ground-truth bounding box is greater than 0.5. ... We report the R@1, R@5, and R@10 metrics, where R@1/5/10 means calculating the recall using the top 1/5/10 generated bounding boxes.

Citation

@misc{peng2023kosmos2,
  title={Kosmos-2: Grounding Multimodal Large Language Models to the World},
  author={Peng et al. (2023)},
  year={2023},
  note={arXiv:2306.14824}
}

• arXiv: 2306.14824