Mac Cvr Eval

mac-cvr-eval

MAC: A Conversion Rate Prediction Benchmark Featuring Labels Under Multiple Attribution Mechanisms — Jinqi Wu et al. (2026) (arXiv:2603.02184, 2026)

What this evaluates

This benchmark evaluates a model's ability to predict conversion rates for ad clicks under multiple attribution mechanisms. It probes ranking capability by measuring how well predicted probabilities distinguish positive from negative samples, both globally and per user. The task treats conversion prediction as a weighted binary classification problem where continuous attribution weights act as sample importance weights.

Datasets

• MAC — total 79000000; splits: (unstated); HF alimamaTech/MAC

Metrics

• AUC — range: [0, 1]
  • Area Under the ROC Curve. Computed as (1/(n+ * n-)) * sum_{i=1}^{n+} sum_{j=1}^{n-} [I(y_i^+ > y_j^-) + 0.5 * I(y_i^+ = y_j^-)], where n+ and n- are counts of positive and negative samples, and y^+ and y^- are predicted probabilities.
• GAUC (primary) — range: [0, 1]
  • Group AUC. Computed as sum_{u=1}^{U} (#click(u) * AUC_u) / sum_{u=1}^{U} #click(u), where U is the number of users, #click(u) is the number of clicks for user u, and AUC_u is the AUC computed on samples from user u. Serves as the primary evaluation criterion.

Input / output format

Input: Categorical features (7 user, 10 item, 3 context), behavior sequence features (user purchase item sequence truncated to max length 20, with item/shop/category IDs and visual similarity scores), and multi-attribution continuous weights for four mechanisms (last-click, first-click, linear, DDA).

Output: A single continuous predicted conversion probability ŷ^{A_t} for the designated target attribution mechanism.

Scoring recipe

def compute_auc(pos_preds, neg_preds):
    ties = sum(1 for p in pos_preds for n in neg_preds if p == n)
    wins = sum(1 for p in pos_preds for n in neg_preds if p > n)
    return (wins + 0.5 * ties) / (len(pos_preds) * len(neg_preds))

def compute_gauc(user_preds, user_labels, user_click_counts):
    gauc_num = 0.0
    total_clicks = 0
    for u in range(len(user_preds)):
        pos = [p for p, l in zip(user_preds[u], user_labels[u]) if l == 1]
        neg = [p for p, l in zip(user_preds[u], user_labels[u]) if l == 0]
        auc_u = compute_auc(pos, neg)
        clicks = user_click_counts[u]
        gauc_num += clicks * auc_u
        total_clicks += clicks
    return gauc_num / total_clicks

Common pitfalls

• Treating continuous attribution weights as hard binary labels without applying them as sample importance weights in the loss or evaluation.
• Computing AUC globally instead of per-user, which fails to capture the user-level ranking alignment that GAUC is designed to measure.
• Confusing auxiliary attribution mechanisms with the target mechanism; only the target mechanism's weights are used for the primary evaluation task.

Evidence (verbatim from paper)

For evaluating model performance, we report the widely-adopted AUC and Group AUC (GAUC) metrics of the primary task. AUC is a widely used ranking metric that evaluates a model’s ability to distinguish positive and negative samples, making it well-suited for CVR prediction. Moreover, we report the GAUC metric, which captures the model’s ability to rank ad clicks within individual user groups and serves as the primary evaluation criterion in our production system.

Citation

@misc{wu2026mac,
  title={MAC: A Conversion Rate Prediction Benchmark Featuring Labels Under Multiple Attribution Mechanisms},
  author={Jinqi Wu et al. (2026)},
  year={2026},
  note={arXiv:2603.02184}
}

• arXiv: 2603.02184