Paco Lvis Instruct Eval

paco-lvis-instruct-eval

SAM3-I: Segment Anything with Instructions — Li et al. (2025) (arXiv:2512.04585, 2025)

What this evaluates

Evaluates a model's ability to follow complex natural-language instructions to segment specific object instances in images. It probes fine-grained instance grounding while maintaining concept-level recall across simple and complex prompts.

Datasets

• PACO-LVIS-Instruct — total 45308; splits: train (-1), val (-1), test (-1)

Metrics

• gIoU (primary) — range: [0, 1]
  • Average Intersection-over-Union (IoU) across all images in the dataset.
• P@50 — range: percent
  • Percentage of test samples where the predicted segmentation mask achieves an IoU of at least 0.5 with the ground-truth mask.

Input / output format

Input: An input image paired with a natural-language instruction specifying the target instance (concept, simple, or complex instruction).

Output: A predicted binary segmentation mask for the target instance specified by the instruction.

Scoring recipe

def compute_metrics(predictions, ground_truths):
    ious = [iou(pred, gt) for pred, gt in zip(predictions, ground_truths)]
    gIoU = sum(ious) / len(ious)
    P50 = sum(1 for i in ious if i >= 0.5) / len(ious) * 100
    return gIoU, P50

Common pitfalls

• Comparing against baselines that use iterative multi-round agent checking (e.g., 3 rounds) rather than single-pass inference, which inflates baseline performance unfairly.
• The instruction-mask pairs are generated via a specific pipeline; evaluating on out-of-distribution instructions without this generation context may yield inconsistent results.
• The paper notes this is a preliminary feasibility study, so results should not be interpreted as state-of-the-art for the broader PIS task.

Evidence (verbatim from paper)

To train and evaluate PIS models, we extend the PACO-LVIS*[paco]* benchmark with rich natural-language instructions, resulting in PACO-LVIS-Instruct. The dataset includes 45,308 images, 210,876 object-level masks, and 843,504 positive instructions, with an average instruction length of 15.3 words. The train/val/test splits follow those of the original PACO-LVIS benchmark, and all instruction–mask pairs are produced using the pipeline introduced in Sec.[5]. For evaluation, we adopt the gIoU and P@50 metrics. Here gIoU [lai2024lisa] is defined by the average of all per-image Intersection-over-Union (IoU). Precision@0.5 (P@50 [hu2016segmentation]) measures the percentage of test samples whose predicted segmentation mask achieves an Intersection-over-Union (IoU) of at least 0.5 with the corresponding ground-truth mask. The percent symbol is omitted for readability.

Citation

@misc{li2025sam3i,
  title={SAM3-I: Segment Anything with Instructions},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2512.04585}
}

• arXiv: 2512.04585