Fashion Compatibility Eval

fashion-compatibility-eval

Learning Type-Aware Embeddings for Fashion Compatibility — Vasileva et al. (2018) (arXiv:1803.09196, 2018)

What this evaluates

Evaluates a model's ability to score the visual-semantic compatibility of items within a complete outfit, and to recommend a missing item that best completes a partial outfit. It probes the model's capacity to learn non-transitive, type-aware relationships across different fashion categories.

Datasets

• Maryland Polyvore — total ?; splits: train (17316), test (3076), validation (1407)
• Polyvore Outfits — total ?; splits: train (-1), test (-1), validation (-1)
• Polyvore Outfits-D — total ?; splits: train (-1), test (-1), validation (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Computed over predicted compatibility scores versus ground truth labels for compatible vs. incompatible outfits.
• accuracy — range: [0, 1]
  • Fraction of correctly identified fill-in-the-blank items out of the total number of questions. Evaluated over 4 candidate items per question.

Input / output format

Input: Outfit represented as a set of item embeddings (visual ResNet-18 features and text HGLMM Fisher vectors). For FITB: an outfit missing one item type, plus four candidate embeddings of that type.

Output: Compatibility task: scalar compatibility score. FITB task: index of the single most compatible candidate item.

Scoring recipe

def score_compatibility(y_true, y_pred):
    return roc_auc_score(y_true, y_pred)

def score_fitb(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds)

Common pitfalls

• Random negative sampling in the original dataset creates invalid outfits (e.g., outfits with only one item type or mismatched categories), making evaluation trivial without category-aware sampling.
• Using unrestricted test negatives without composition filtering artificially inflates performance, as models can dismiss invalid compositions without learning true compatibility.

Evidence (verbatim from paper)

In the fashion compatibility task, a candidate outfit is scored as to whether its constitute items are compatible with each other. Performance is evaluated using the average under a receiver operating characteristic curve (AUC). The second task is to select from a set of candidate items (four in this case) in a fill-in-the-blank (FITB) fashion recommendation experiment. The goal is to select the most compatible item with the remainder of the outfit, and performance is evaluated by accuracy on the answered questions.

Citation

@misc{vasileva2018learning,
  title={Learning Type-Aware Embeddings for Fashion Compatibility},
  author={Vasileva et al. (2018)},
  year={2018},
  note={arXiv:1803.09196}
}

• arXiv: 1803.09196