Fashion Ner El Eval

fashion-ner-el-eval

Enriching a Fashion Knowledge Graph from Product Textual Descriptions — Barroca et al. (2022) (arXiv:2206.01087, 2022)

What this evaluates

Evaluates a BERT-based Named Entity Recognition pipeline and a binary classifier for candidate entity disambiguation on fashion product descriptions. It probes the model's ability to extract attribute mentions (e.g., material, color) and correctly link them to a knowledge graph ontology under severe data scarcity.

Datasets

• Fashion Product Descriptions (In-house) — total 900; splits: train (720), val (180)
• Fashion EL Disambiguation Dataset — total 2528; splits: train (-1), test (-1)

Metrics

• f1-score (primary) — range: [0, 1]
  • Standard binary classification F1-score, calculated as the harmonic mean of precision and recall. Accuracy is explicitly excluded due to severe class imbalance (7.2% positive samples).

Input / output format

Input: NER: Raw product textual description. EL: Product textual description, attribute mention span, and candidate entity string.

Output: NER: Token-level entity type tags (e.g., BIO format). EL: Binary label (1 for correct link, 0 for incorrect).

Scoring recipe

def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0

Common pitfalls

• Accuracy is explicitly not reported due to high class imbalance (7.2% positive), so readers should not use or expect accuracy as the evaluation metric.
• The EL dataset is highly synthetic, generated from only 32 manually annotated instances via fuzzy matching, which may limit generalization to out-of-distribution product queries.

Evidence (verbatim from paper)

As metrics, we used the standard metrics for binary classification: precision, recall and f1-score. We do not report the accuracy score due to the high unbalanced nature of the dataset.

Citation

@misc{barroca2022enriching,
  title={Enriching a Fashion Knowledge Graph from Product Textual Descriptions},
  author={Barroca et al. (2022)},
  year={2022},
  note={arXiv:2206.01087}
}

• arXiv: 2206.01087