Esci Similarity And Token Class Eval

esci-similarity-and-token-class-eval

RexBERT: Context Specialized Bidirectional Encoders for E-commerce — Bajaj et al. (2026) (arXiv:2602.04605, 2026)

What this evaluates

Evaluates e-commerce language understanding through masked token recovery on product texts and graded semantic similarity between search queries and products. Also assesses general natural language understanding capabilities via the GLUE benchmark.

Datasets

• Amazon ESCI — total 2600000; splits: train (-1), test (-1)
• GLUE — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• top-k accuracy (primary) — range: [0, 1]
  • Proportion of instances where the ground-truth token appears in the model's top-k predicted tokens. Evaluated at k=1, 3, and 5.
• Spearman correlation (primary) — range: [-1, 1]
  • Rank correlation between predicted cosine similarities and mapped ESCI relevance scores (Exact=1.0, Substitute=0.66, Complement=0.33, Irrelevant=0.0).
• accuracy — range: [0, 1]
  • Standard classification accuracy for GLUE tasks (SST-2, QNLI, QQP, MNLI).
• Pearson/Spearman correlation — range: [-1, 1]
  • Standard correlation metric used for the STS-B task in GLUE.

Input / output format

Input: Token classification: product titles/descriptions truncated to 128, 256, or 512 tokens with 15% span-aware masking. Semantic similarity: query-product text pairs (title concatenated with description, truncated to context window). GLUE: standard sentence or sentence-pair inputs per task.

Output: Token classification: ranked list of predicted tokens (top-k). Semantic similarity: scalar cosine similarity score. GLUE: class label or regression score.

Scoring recipe

def topk_accuracy(preds, gold, k):
    return sum(1 for p, g in zip(preds, gold) if g in p[:k]) / len(gold)

def spearman_esci(pred_sims, gold_labels):
    score_map = {'Exact': 1.0, 'Substitute': 0.66, 'Complement': 0.33, 'Irrelevant': 0.0}
    target_scores = [score_map[l] for l in gold_labels]
    return spearmanr(pred_sims, target_scores).correlation

def glue_metric(preds, gold, task):
    if task in ['SST-2', 'QNLI', 'QQP', 'MNLI']:
        return accuracy_score(gold, preds)
    else:  # STS-B
        return pearsonr(gold, preds).correlation

Common pitfalls

• ESCI uses a 4-level graded relevance scale (1.0, 0.66, 0.33, 0.0) rather than binary labels; using binary accuracy will misrepresent performance.
• Token classification uses span-aware masking (15% of tokens), not standard random token masking.
• Evaluation must be reported separately for 128, 256, and 512 token truncation lengths.
• GLUE evaluation uses development sets, not test sets, for fair comparison with baselines.

Evidence (verbatim from paper)

We map the four-level ESCI relevance labels to numerical scores: 1.0 for Exact, 0.66 for Substitute, 0.33 for Complement and 0.0 for Irrelevant. Models are fine-tuned using the CoSENT loss*(Huang et al., [2024])*, which optimises pairwise orderings in cosine similarity space and has been shown to reduce anisotropy in BERT embeddings while producing more consistent similarity rankings. We report Spearman’s rank correlation between predicted cosine similarities and target scores on a held-out set, which evaluates whether the embedding space preserves the ordinal structure of the ESCI labels.

Citation

@misc{bajaj2026rexbert,
  title={RexBERT: Context Specialized Bidirectional Encoders for E-commerce},
  author={Bajaj et al. (2026)},
  year={2026},
  note={arXiv:2602.04605}
}

• arXiv: 2602.04605