Trace Gmv Prediction Eval

trace-gmv-prediction-eval

Delayed Feedback Modeling for Post-Click Gross Merchandise Volume Prediction: Benchmark, Insights and Approaches — Li et al. (2026) (arXiv:2601.20307, 2026)

What this evaluates

This benchmark evaluates a model's ability to predict post-click Gross Merchandise Volume (GMV) under delayed feedback conditions. It specifically probes how well models adapt to rapidly evolving label distributions through online streaming training and whether they can effectively handle the distinct statistical properties of single-purchase versus repurchase transactions.

Datasets

• TRACE — total 7160000; splits: train (-1), test (-1); repo https://github.com/alimama-tech/OnlineGMV

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen positive instance (click leading to GMV) is ranked higher than a negative instance. Commonly used in ad tech to evaluate the ranking quality of predicted GMV scores.

Input / output format

Input: A tuple ⟨x, t, P⟩ containing: (1) x ∈ R^d, an embedding vector of 22 categorical user/item/context features; (2) t, the click timestamp; (3) P = {(t_i^p, p_i)}_{i=1}^N, a sequence of N purchases within a 7-day attribution window, where each element contains the purchase timestamp t_i^p and transaction price p_i.

Output: A single scalar value ŷ representing the predicted post-click GMV.

Scoring recipe

def compute_auc(predictions, labels):
    # predictions: array of predicted GMV scores
    # labels: array of binary indicators (1 if GMV > 0, else 0)
    order = np.argsort(predictions)[::-1]
    sorted_labels = labels[order]
    tp = 0
    fp = 0
    auc = 0.0
    prev_score = None
    for i, label in enumerate(sorted_labels):
        if prev_score is not None and predictions[order[i]] != prev_score:
            auc += (tp + fp) * (i - tp)
            prev_score = predictions[order[i]]
        if label == 1:
            tp += 1
        else:
            fp += 1
    auc += (tp + fp) * (len(sorted_labels) - tp)
    num_pos = np.sum(labels == 1)
    num_neg = np.sum(labels == 0)
    return auc / (num_pos * num_neg) if (num_pos > 0 and num_neg > 0) else 0.0

Common pitfalls

• Using only the final ground-truth GMV label y* for offline training ignores the delayed feedback nature of the task and fails to capture intra-day label distribution shifts.
• Directly using partially observed GMV y^(t) as the training target causes severe underestimation bias because a large portion of GMV arrives after the observation point.
• Training a single unified model on all samples ignores the significant distributional discrepancy between single-purchase and repurchase samples, hindering optimization.

Evidence (verbatim from paper)

As the AUC metrics listed in Table[3], the online-trained model significantly outperforms the daily-updated offline counterpart, which substantiates the significance of model freshness and the superiority of online training.

Citation

@misc{li2026trace,
  title={Delayed Feedback Modeling for Post-Click Gross Merchandise Volume Prediction: Benchmark, Insights and Approaches},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2601.20307}
}

• arXiv: 2601.20307