Luss Eval

luss-eval

Large-scale Unsupervised Semantic Segmentation — Gao et al. (2021) (arXiv:2106.03149, 2021)

What this evaluates

Evaluates the ability of models to perform pixel-level semantic segmentation on large-scale, diverse image collections without human annotations. It probes unsupervised representation learning, category discovery, and fine-grained mask prediction capabilities.

Datasets

• ImageNet-S — total ?; splits: val (-1), test (-1)
• ImageNet-S50 — total ?; splits: val (-1), test (-1)
• ImageNet-S300 — total ?; splits: val (-1), test (-1)

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union across all semantic classes. Calculated as the average of IoU per class, where IoU is the intersection over union of predicted and ground truth masks for each class.
• b-mIoU — range: [0, 1]
  • Boundary mean Intersection over Union. Similar to mIoU but computed on predicted and ground truth object boundaries rather than full masks.
• Img-Acc — range: [0, 1]
  • Image-level accuracy. The percentage of images where the predicted segmentation mask matches the ground truth mask exactly or meets a specified threshold.
• Fβ — range: [0, 1]
  • F-beta score balancing precision and recall for segmentation masks, typically weighted towards recall (β > 1) to penalize missed objects.

Input / output format

Input: RGB images (evaluated at full resolution; 224x224 crops used during training)

Output: Pixel-wise class assignment masks (segmentation maps) for each input image

Scoring recipe

def compute_miou(pred_masks, gt_masks, num_classes):
    intersection = np.zeros(num_classes)
    union = np.zeros(num_classes)
    for p, g in zip(pred_masks, gt_masks):
        for c in range(num_classes):
            intersection[c] += np.logical_and(p == c, g == c).sum()
            union[c] += np.logical_or(p == c, g == c).sum()
    iou = intersection / np.maximum(union, 1)
    return np.mean(iou)

Common pitfalls

• By default, the 'other' category is included in mIoU and b-mIoU calculations; excluding it changes scores significantly.
• Performance is highly sensitive to object size; small objects yield lower mIoU and b-mIoU, so aggregate scores may hide scale-specific failures.
• Fully unsupervised protocol strictly forbids supervised ImageNet-1k pre-training; methods using it are not directly comparable.

Evidence (verbatim from paper)

Comparison of our proposed LUSS method and existing USS methods on the ImageNet-S dataset using the fully unsupervised evaluation protocol. Test mIoU under different object sizes are provided. ... By default, the 'other' category is used to calculate mIoU and b-mIoU.

Citation

@misc{gao2021large,
  title={Large-scale Unsupervised Semantic Segmentation},
  author={Gao et al. (2021)},
  year={2021},
  note={arXiv:2106.03149}
}

• arXiv: 2106.03149