Instructpart Eval

instructpart-eval

InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning — Wan et al. (2025) (arXiv:2505.18291, 2025)

What this evaluates

Evaluates Vision-Language Models' ability to perform fine-grained visual grounding and instruction reasoning for part segmentation. It probes whether models can infer task-relevant object parts from natural language instructions or oracle prompts, and assesses their capacity for affordance learning in human-robot interaction contexts.

Datasets

• InstructPart — total 2400; splits: train (-1), test (-1)

Metrics

• gIoU (primary) — range: percent
  • Average of all per-image Intersection-over-Union (IoU) scores across the dataset.
• cIoU — range: percent
  • Cumulative intersection over cumulative union across all images in the dataset.
• P@50 — range: percent
  • Precision metric where a prediction is considered a true positive if the IoU ratio exceeds 0.5.
• P@50:95 — range: percent
  • Precision averaged across IoU thresholds from 0.50 to 0.95 in increments of 0.05.

Input / output format

Input: RGB image paired with a natural language instruction (either an oracle referring prompt containing explicit object/part names, or a task-oriented reasoning prompt requiring implicit part inference).

Output: Binary segmentation mask corresponding to the specified part.

Scoring recipe

def compute_metrics(pred_masks, gt_masks):
    ious = [np.sum(p & g) / np.sum(p | g) for p, g in zip(pred_masks, gt_masks)]
    gIoU = np.mean(ious) * 100
    cIoU = (np.sum([np.sum(p & g) for p, g in zip(pred_masks, gt_masks)]) /
            np.sum([np.sum(p | g) for p, g in zip(pred_masks, gt_masks)])) * 100
    p50 = np.mean([1.0 if iou > 0.5 else 0.0 for iou in ious]) * 100
    p50_95 = np.mean([np.mean([1.0 if iou > t else 0.0 for t in np.arange(0.50, 0.96, 0.05)]) for iou in ious]) * 100
    return gIoU, cIoU, p50, p50_95

Common pitfalls

• Confusing gIoU (per-image average) with cIoU (dataset-level cumulative intersection/union), which can yield significantly different values on imbalanced datasets.
• P@50:95 requires averaging precision across multiple IoU thresholds (0.50 to 0.95 step 0.05), not just evaluating at a single threshold.
• GPT-4V cannot output masks directly; the paper's pipeline uses Grounding-DINO + SAM, so poor results may stem from the grounding/masking step rather than the VLM's reasoning.

Evidence (verbatim from paper)

To evaluate our approach, we use standard metrics in LISA, namely gIoU and cIoU. gIoU reflects the average of all per-image Intersection-over-Unions (IoUs), while cIoU is defined by the cumulative intersection over the cumulative union. To evaluate the precision of the models, we adopt Precision@50 (P@50) metric as the previous referring segmentation works and develop a Precision@50:95 (P@50:95) metric according to COCO. The P@50 metric considers a mask to be a true positive when the IoU ratio exceeds 0.5, and P@50:95 calculates across a range of IoU thresholds from 0.50 to 0.95 with increments of 0.05, then averages across all the thresholds.

Citation

@misc{wan2025instructpart,
  title={InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning},
  author={Wan et al. (2025)},
  year={2025},
  note={arXiv:2505.18291}
}

• arXiv: 2505.18291