Fashionpedia Eval

fashionpedia-eval

Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset — Jia et al. (2020) (arXiv:2004.12276, 2020)

What this evaluates

Evaluates joint instance segmentation and fine-grained attribute localization on fashion apparel. It measures how well a model can detect objects, segment them accurately, and correctly assign multiple localized attributes to each instance.

Datasets

• Fashionpedia — total 48825; splits: train (-1), val (-1), test (-1); repo https://fashionpedia.github.io/home/Model_and_API.html

Metrics

• AP_{IoU + F_1} (primary) — range: percent
  • Average Precision averaged across 10 IoU thresholds (0.5 to 0.95) and 10 macro F1 thresholds (0.5 to 0.95). A detection is a true positive only if its bounding box IoU exceeds the IoU threshold AND the macro F1 score of its predicted attributes exceeds the F1 threshold.
• AP_{IoU} — range: percent
  • Standard COCO Average Precision averaged across 10 IoU thresholds (0.5 to 0.95) and all categories, ignoring attribute constraints.

Input / output format

Input: RGB images resized to 1024px (or 1280px for SpineNet-143) containing fashion apparel. Ground truth provides per-instance bounding boxes, segmentation masks, and multi-label attribute annotations.

Output: Per-instance predictions consisting of bounding boxes, binary segmentation masks, and a multi-label vector of predicted attributes.

Scoring recipe

aps = []
for cat in categories:
    for tau_iou in np.arange(0.5, 1.0, 0.05):
        for tau_f1 in np.arange(0.5, 1.0, 0.05):
            tps = 0
            for pred, gt in matches(preds[cat], gts[cat]):
                if pred.iou(gt) >= tau_iou and pred.macro_f1(gt) >= tau_f1:
                    tps += 1
            aps.append(compute_ap(tps, len(gts[cat])))
return np.mean(aps)

Common pitfalls

• Joint segmentation and attribute localization causes a significant performance drop compared to segmentation-only baselines due to the strict F1 constraint.
• Standard COCO detectors fail on Fashionpedia due to domain shift, showing high background confusion and localization errors.
• Macro F1 thresholding requires precise mask alignment; coarse masks artificially lower attribute F1 scores.

Evidence (verbatim from paper)

In the case of instance segmentation and attribute localization, we extend standard COCO metric by adding one more constraint: the macro F1 score for predicted attributes of single detected object with category c (see supplementary material for the average choice of f1-score). We denote the F1 threshold as τF1, and it has the same range as τIoU (τF1 ∈ [0.5 : 0.05 : 0.95]). The main metric AP_{IoU + F1} reports averaged precision score across all 10 IoU thresholds, all 10 macro F1 scores, and all the categories.

Citation

@misc{jia2020fashionpedia,
  title={Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset},
  author={Jia et al. (2020)},
  year={2020},
  note={arXiv:2004.12276}
}

• arXiv: 2004.12276