Padt Unified Vision Eval

padt-unified-vision-eval

Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs — Su et al. (2025) (arXiv:2510.01954, 2025)

What this evaluates

Evaluates a multimodal large language model's ability to perform visual grounding, segmentation, open-vocabulary detection, and referring image captioning by predicting structured visual outputs directly from interleaved visual reference tokens and text.

Datasets

• RefCOCO/+/g — total ?; splits: val (-1), test-A (-1), test-B (-1), test (-1)
• COCO 2017 — total ?; splits: val (-1)
• RIC — total ?; splits: val (-1)

Metrics

• IoU@0.5 accuracy (primary) — range: percent
  • Accuracy is computed as the fraction of predictions where the Intersection over Union (IoU) between the predicted bounding box and the ground-truth box exceeds 0.5.
• cIoU — range: [0, 1]
  • centroid IoU, measuring the overlap between predicted and ground-truth segmentation masks while penalizing centroid displacement.
• AP@[50:95] — range: [0, 1]
  • Average Precision averaged over IoU thresholds from 0.50 to 0.95 in steps of 0.05, standard for COCO object detection evaluation.
• CIDEr-D, Meteor, ROUGE-L, BLEU-4, GP, GR — range: [0, 1] | percent
  • Standard NLP metrics for caption quality (CIDEr-D, Meteor, ROUGE-L, BLEU-4) combined with GreedyPrecision (GP) and GreedyRecall (GR) for grounding accuracy in referring image captioning.

Input / output format

Input: An image and a natural language query/prompt describing the target object or scene.

Output: A sequence of Visual Reference Tokens (VRTs) interleaved with text, which are decoded into bounding boxes, segmentation masks, or descriptive captions.

Scoring recipe

def score_rec(pred_box, gt_box):
    iou = compute_iou(pred_box, gt_box)
    return 1.0 if iou > 0.5 else 0.0

def score_res(pred_mask, gt_mask):
    return compute_centroid_iou(pred_mask, gt_mask)

def score_coco(pred_boxes, gt_boxes):
    return coco_eval.compute_ap(pred_boxes, gt_boxes, iou_thresh=[0.50, 0.55, ..., 0.95])

def score_ric(pred_caption, gt_captions, gt_caption):
    return {
        'CIDEr-D': cider_score(pred_caption, gt_captions),
        'Meteor': meteor_score(pred_caption, gt_captions),
        'ROUGE-L': rouge_l_score(pred_caption, gt_caption),
        'BLEU-4': bleu4_score(pred_caption, gt_caption),
        'GP': greedy_precision(pred_caption, gt_caption),
        'GR': greedy_recall(pred_caption, gt_caption)
    }

Common pitfalls

• Using coordinate-based text representations instead of the paper's Visual Reference Tokens (VRTs) will break the unified paradigm and yield significantly lower performance.
• Evaluating on RefCOCOg requires using the official 'test' split rather than 'val' to match reported numbers, as test-A/B splits are specific to RefCOCO/RefCOCO+.
• For COCO open-vocabulary detection, models must predict bounding boxes without relying on closed-set class labels; using fixed class priors inflates AP scores.

Evidence (verbatim from paper)

The Referring Expression Comprehension (REC) task evaluates an MLLM’s ability to localize objects given natural language descriptions, where a prediction is considered correct if its IoU with the ground-truth box exceeds 50%. We adopt cIoU as the evaluation metric, and results are reported in Tab.[2]. PaDT and PaDT Pro (3B) deliver strong improvements, reaching 1.45 CIDEr, 0.304 Meteor, 0.501 ROUGE-L, 0.467 BLEU-4, and top detection scores of 82.3% GreedyPrecision (GP) and 45.1% GreddyRecall (GR).

Citation

@misc{su2025patchasdecodabletoken,
  title={Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs},
  author={Su et al. (2025)},
  year={2025},
  note={arXiv:2510.01954}
}

• arXiv: 2510.01954