Visual Semantic Segmentation Eval

visual-semantic-segmentation-eval

Visual Semantic Information Pursuit: A Survey — Daqi Liu et al. (2019) (arXiv:1903.05434, 2019)

What this evaluates

Evaluates a model's ability to assign a semantic class label to every pixel in an image, capturing fine-grained scene understanding. It measures pixel-level classification accuracy and boundary alignment across diverse outdoor and indoor environments.

Datasets

• Pascal Context — total 10103; splits: train (4998), test (5105)
• Sift Flow — total 2688; splits: test (2688)
• COCO Stuff — total 10000; splits: train (9000), test (1000)

Metrics

• GPA (primary) — range: [0, 1]
  • Global Pixel Accuracy: percentage of all pixels correctly classified across the entire image.
• ACA — range: [0, 1]
  • Average per-Class Accuracy: mean of pixel accuracy computed independently for each semantic class.
• mIOU — range: [0, 1]
  • mean Intersection of Union: mean of IoU scores computed per class, measuring overlap between predicted and ground-truth masks.

Input / output format

Input: RGB image.

Output: Pixel-wise segmentation map with class labels for each pixel.

Scoring recipe

def compute_seg_metrics(pred_map, gt_map, classes):
    total_pixels = pred_map.size
    correct = (pred_map == gt_map).sum()
    gpa = correct / total_pixels
    acs = []
    ious = []
    for c in classes:
        pred_c = (pred_map == c)
        gt_c = (gt_map == c)
        acs.append((pred_c & gt_c).sum() / gt_c.sum())
        ious.append((pred_c & gt_c).sum() / (pred_c | gt_c).sum())
    aca = sum(acs) / len(acs)
    miou = sum(ious) / len(ious)
    return gpa, aca, miou

Common pitfalls

• GPA is heavily biased towards dominant background classes; ACA or mIOU are better for imbalanced datasets.
• mIOU calculation varies on whether background/stuff classes are included; papers must specify class set.
• Sift Flow and Pascal Context use different class taxonomies (33 vs 540 classes), making cross-dataset comparison difficult.

Evidence (verbatim from paper)

To evaluate the performances of the visual semantic segmentation methods, three main metrics are generally applied in the existing literatures: Global Pixel Accuracy (GPA), Average per-Class Accuracy (ACA) and mean Intersection of Union (mIOU). Specifically, GPA represents the percentage of all correctly classified pixels, ACA depicts the mean of class-wise pixel accuracy and mIOU is the mean of the accuracy metric IOU.

Citation

@misc{liu2019visualsemantic,
  title={Visual Semantic Information Pursuit: A Survey},
  author={Daqi Liu et al. (2019)},
  year={2019},
  note={arXiv:1903.05434}
}

• arXiv: 1903.05434