Yahoo R6b Ctr Eval

yahoo-r6b-ctr-eval

Adversarial Gradient Driven Exploration for Deep Click-Through Rate Prediction — Wu et al. (2021) (arXiv:2112.11136, 2021)

What this evaluates

This evaluation probes a model's ability to optimize exploration-exploitation trade-offs in personalized click-through rate (CTR) prediction. It measures how effectively an exploration strategy improves cumulative user engagement and advertiser retention compared to standard ranking baselines.

Datasets

• Yahoo! R6B — total 28000000; splits: warmup (80000), evaluation (-1)

Metrics

• CTR (primary) — range: percent
  • Click-through rate, calculated as the total number of user clicks divided by the total number of impressions. Higher values indicate better recommendation performance.
• PCOC — range: [0, 1]
  • Predicted CTR over the true CTR. It measures calibration between predicted scores and actual click rates, with a target value closer to 1 indicating better alignment.
• AFR — range: percent
  • Advertiser Follow-up Rate, measuring the proportion of advertisers willing to renew their contracts with the platform after exposure to the recommendation strategy.
• Total Clicks — range: other
  • The sum of binary click labels (0/1) over the evaluation period, used as an offline approximation for cumulative rewards in exploration strategy assessment.

Input / output format

Input: User features (e.g., 136-dimensional multi-hot vectors for demographics), a list of candidate article IDs (typically ~38 per visit), and historical interaction context.

Output: A predicted CTR probability score for each candidate article, used to rank items for display.

Scoring recipe

def score(preds, gold, impressions):
    total_clicks = sum(gold)
    ctr = total_clicks / impressions
    true_ctr = total_clicks / impressions
    pred_ctr = sum(preds) / impressions
    pcoc = pred_ctr / true_ctr if true_ctr > 0 else 0.0
    return {'total_clicks': total_clicks, 'ctr': ctr, 'pcoc': pcoc}

Common pitfalls

• Offline replay evaluation using historical data cannot capture real-time feedback-loop amplification or online learning dynamics present in production.
• PCOC is frequently inverted by readers; the paper explicitly defines it as predicted CTR divided by true CTR, where 1 is optimal.
• The 80,000-sample time-based warmup split may introduce temporal bias if the evaluation period exhibits significant distribution shifts.

Evidence (verbatim from paper)

For the offline evaluation, we utilize the total number of user clicks as an approximation for the cumulative rewards. This aligns with most of the previous studies, in which user click is often treated as exploration pay-off in personalized recommnder systems. A large number of user clicks usually indicate a better performance for an exploration strategy. With respect to the online A/B testing, we employ several standard metrics such as click-through rate (CTR) and prediction accuracy (e.g. PCOC) for evaluation. Here, PCOC (predicted CTR over the true CTR) examines whether the predictive score aligns with the actual click rate. For this metric, our goal is to obtain a value that is closer to 1. In the context of online advertising, we also evaluate the exploration strategy with a top-line business metric named AFR (Advertiser Follow-up Rate), which measures the willingness of an advertiser to renew its contract with our platform.

Citation

@misc{wu2021adversarial,
  title={Adversarial Gradient Driven Exploration for Deep Click-Through Rate Prediction},
  author={Wu et al. (2021)},
  year={2021},
  note={arXiv:2112.11136}
}

• arXiv: 2112.11136