Contextual Object Detection Eval

contextual-object-detection-eval

Contextual Object Detection with Multimodal Large Language Models — Zang et al. (2023) (arXiv:2305.18279, 2023)

What this evaluates

Probes a multimodal large language model's ability to infer and localize objects within human-AI interaction contexts (e.g., cloze tests, captioning, QA) using open-vocabulary object names, rather than fixed class sets.

Datasets

• CODE — total 708863; splits: train (665161), val (22061), test (21641); repo https://github.com/yuhangzang/ContextDET

Metrics

• Acc@1 (primary) — range: percent
  • Percentage of correctly predicted object words in the cloze test setting, where the model generates a single top-1 word to fill a masked token in a caption.
• Acc@5 — range: percent
  • Percentage of cases where the ground-truth object word appears in the model's top-5 predicted words, used to account for synonym variation.
• AP@1 — range: percent
  • Mean Average Precision computed over bounding boxes predicted using the model's top-1 object word.
• AP@5 — range: percent
  • Mean Average Precision computed over bounding boxes predicted using the model's top-5 object words.

Input / output format

Input: An image, a caption with a masked object name (e.g., 'A [MASK] is sitting on the grass'), and optionally a question or context for QA/captioning settings.

Output: A predicted object word (top-1 or top-5) and corresponding bounding box coordinates for the detected object.

Scoring recipe

def score(predictions, golds):
    # Text accuracy
    correct = sum(1 for p, g in zip(predictions, golds) if g in p[:5])
    acc = (correct / len(golds)) * 100
    
    # Box AP (standard COCO-style mAP with IoU=0.5)
    # AP@1 uses boxes paired with top-1 predictions
    # AP@5 uses boxes paired with top-5 predictions
    ap1 = compute_mAP(predictions_boxes_top1, gold_boxes, iou_thresh=0.5)
    ap5 = compute_mAP(predictions_boxes_top5, gold_boxes, iou_thresh=0.5)
    return acc, ap1, ap5

Common pitfalls

• Evaluating on fixed class IDs instead of flexible object name words, which defeats the open-vocabulary purpose.
• Ignoring top-5 metrics (Acc@5, AP@5), which are crucial due to synonym variation and fine-grained vocabulary challenges.
• Assuming standard closed-set detection; the task requires generating language first before detecting boxes.

Evidence (verbatim from paper)

In our contextual cloze test setting, we compute accuracy by calculating the percentage of correctly predicted object words. However, evaluating this accuracy poses a challenge due to the presence of numerous synonyms and fine-grained object words in human language, which can be difficult for annotators to distinguish. This is a problem similar to those faced by previous large vocabulary image-classification datasets, such as ImageNet, which use the top-5 accuracy metric as a supplementary metric to the top-1 accuracy. Consequently, we also adopt both the top-1 accuracy (Acc@1) and the top-5 accuracy (Acc@5) as our evaluation metrics. For box evaluation, we compute the mean Average Precision (mAP) metric based on the top-1 and top-5 predicted names, which are represented as AP@1 and AP@5.

Citation

@misc{zang2023contextdet,
  title={Contextual Object Detection with Multimodal Large Language Models},
  author={Zang et al. (2023)},
  year={2023},
  note={arXiv:2305.18279}
}

• arXiv: 2305.18279