Reasonseg Eval

reasonseg-eval

LISA: Reasoning Segmentation via Large Language Model — Xin Lai et al. (arXiv:2308.00692, 2023)

What this evaluates

Evaluates a model's ability to generate precise segmentation masks from implicit, complex text queries that require reasoning and world knowledge. It specifically probes whether the model can move beyond simple explicit referring expressions to handle multi-step logical deductions and visual grounding simultaneously.

Datasets

• ReasonSeg — total ?; splits: val (-1), test (-1); repo https://github.com/dvlab-research/LISA
• refCOCO — total ?; splits: val (-1), testA (-1), testB (-1)
• refCOCO+ — total ?; splits: val (-1), testA (-1), testB (-1)
• refCOCOg — total ?; splits: val(U) (-1), test(U) (-1)

Metrics

• gIoU (primary) — range: percent
  • Average of all per-image Intersection-over-Union (IoU) scores across the dataset.
• cIoU — range: percent
  • Cumulative intersection over the cumulative union of predicted and ground-truth masks across all images.

Input / output format

Input: An image and a natural language query (often implicit or requiring multi-step reasoning).

Output: A binary segmentation mask corresponding to the queried object(s) in the image.

Scoring recipe

def compute_giou(pred_masks, gt_masks):
    ious = []
    for pred, gt in zip(pred_masks, gt_masks):
        intersection = (pred & gt).sum()
        union = (pred | gt).sum()
        ious.append(intersection / union if union > 0 else 0.0)
    return sum(ious) / len(ious)

def compute_ciou(pred_masks, gt_masks):
    total_intersection = sum((p & g).sum() for p, g in zip(pred_masks, gt_masks))
    total_union = sum((p | g).sum() for p, g in zip(pred_masks, gt_masks))
    return total_intersection / total_union if total_union > 0 else 0.0

Common pitfalls

• cIoU is highly biased toward large-area objects and fluctuates significantly, making gIoU the preferred metric for fair comparison.
• The task requires models to understand implicit queries and leverage world knowledge, so standard referring segmentation models without reasoning capabilities will fail.
• Data leakage must be strictly avoided: COCO samples present in refCOCO validation sets should be excluded during training to ensure evaluation integrity.

Evidence (verbatim from paper)

We follow most previous works on referring segmentation to adopt two metrics: gIoU and cIoU. gIoU is defined by the average of all per-image Intersection-over-Union (IoUs), while cIoU is defined by the cumulative intersection over the cumulative union. Since cIoU is highly biased toward large-area objects and it fluctuates too much, gIoU is preferred.

Citation

@misc{lai2023lisa,
  title={LISA: Reasoning Segmentation via Large Language Model},
  author={Xin Lai et al.},
  year={2023},
  note={arXiv:2308.00692}
}

• arXiv: 2308.00692