Forge Sid Eval

forge-sid-eval

FORGE: Forming Semantic Identifiers for Generative Retrieval in Industrial Datasets — Fu et al. (2025) (arXiv:2509.20904, 2025)

What this evaluates

Evaluates the quality of generated Semantic Identifiers (SIDs) for generative retrieval in industrial recommendation and search systems. It measures how well SIDs capture item relationships and distribute usage fairly, and assesses their impact on downstream retrieval hitrate and online transaction metrics.

Datasets

• FORGE — total 250000000; splits: S1 (-1), S2 (-1), S3 (-1), val (-1); repo https://github.com/selous123/al_sid

Metrics

• HR@K (primary) — range: percent
  • Hit Rate at K (K ∈ {20, 100, 500, 1000}). Calculated as the fraction of queries where the ground-truth item appears in the top-K retrieved items. Reported as a percentage in tables.
• embedding hitrate — range: [0, 1]
  • Measures the quality of item collaborative relationships within the multi-modal feature space by checking if relevant items appear in the top-K nearest neighbors of the query embedding.
• Gini coefficient — range: [0, 1]
  • Statistical measure of inequality in SID usage distribution across items. Lower values indicate a fairer, more balanced assignment of items to SIDs.

Input / output format

Input: Multimodal item features (images, text, categories, sellers), item-to-item co-occurrence data, and user query/context. For SID generation, the model receives item features to produce discrete codebook tokens. For retrieval, it receives a query to predict SIDs and rank items.

Output: Discrete Semantic Identifiers (SIDs) structured as multi-level codebooks (e.g., 3×8192 or 2×32768 tokens per item). For retrieval tasks, the model outputs a ranked list of items or predicted SIDs.

Scoring recipe

def compute_hr_at_k(retrieved_list, ground_truth, k):
    return 1.0 if ground_truth in retrieved_list[:k] else 0.0

def compute_gini(values):
    sorted_vals = np.sort(values)
    n = len(sorted_vals)
    idx = np.arange(1, n + 1)
    return ((2 * np.sum(idx * sorted_vals)) / (n * np.sum(sorted_vals))) - ((n + 1) / n)

# Aggregate over dataset
total_hr = sum(compute_hr_at_k(preds[i], gold[i], k) for i in range(N)) / N
gini_val = compute_gini(sid_usage_counts)

Common pitfalls

• Confusing offline proxy metrics (embedding hitrate, Gini) with online retrieval metrics (HR@K, PVR). The paper explicitly states these direct metrics can reliably proxy GR performance without full model training.
• Assuming SID collision strategies (KNN, Random, i2i) are universally optimal. Performance varies significantly across training stages (S1-S3) and codebook configurations (2-level vs 3-level).
• Misinterpreting HR@K values as raw counts rather than percentages. The paper reports all HR@K results as percentages (e.g., 3.61% for HR@20).

Evidence (verbatim from paper)

The direct evaluations of SIDs using embedding hitrate and Gini coefficient are presented in Figure 3. As we mentioned in Section 3.4, the embedding hitrate captures the quality of item collaborative relationships within the multi-modal feature $\mathcal{H}^{i}$, while the Gini coefficient reflects the fairness of SID distribution.

Citation

@misc{fu2025forge,
  title={FORGE: Forming Semantic Identifiers for Generative Retrieval in Industrial Datasets},
  author={Fu et al. (2025)},
  year={2025},
  note={arXiv:2509.20904}
}

• arXiv: 2509.20904