Wide Deep Recommender Eval

wide-deep-recommender-eval

Wide & Deep Learning for Recommender Systems — Cheng et al. (2016) (arXiv:1606.07792, 2016)

What this evaluates

Evaluates a hybrid recommender model's ability to balance memorization of frequent user-item interactions with generalization to unseen combinations for ranking candidate apps. The protocol measures predictive accuracy on a static holdout set and business impact via live A/B testing on user acquisition rates.

Datasets

• Google Play App Store (internal) — total ?; splits: test (-1)

Metrics

• Offline AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve computed on a holdout set of user-app interactions. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
• Online Acquisition Gain (primary) — range: percent
  • Relative percentage improvement in app acquisition rate compared to the control group in a live A/B test. Calculated as (exp_rate - control_rate) / control_rate * 100%.

Input / output format

Input: Sparse user and item features, including cross-product transformations and dense embeddings, representing candidate apps to be scored for ranking.

Output: A relevance score or probability for each candidate app, used to generate a ranked list of recommendations.

Scoring recipe

def compute_auc(y_true, y_scores):
    return roc_auc_score(y_true, y_scores)

def compute_acquisition_gain(acquisitions_control, users_control, acquisitions_exp, users_exp):
    control_rate = sum(acquisitions_control) / len(users_control)
    exp_rate = sum(acquisitions_exp) / len(users_exp)
    return (exp_rate - control_rate) / control_rate * 100

Common pitfalls

• Offline AUC may not correlate strongly with online business metrics because offline data has fixed impressions and labels, whereas the online system dynamically explores and learns from new user responses.
• Online Acquisition Gain is relative to the specific control model (wide-only logistic regression); gains may not generalize to other baselines or traffic distributions.

Evidence (verbatim from paper)

Besides online experiments, we also show the Area Under Receiver Operator Characteristic Curve (AUC) on a holdout set offline. While Wide & Deep has a slightly higher offline AUC, the impact is more significant on online traffic. ... Wide & Deep model improved the app acquisition rate on the main landing page of the app store by +3.9% relative to the control group (statistically significant).

Citation

@misc{cheng2016widedeep,
  title={Wide & Deep Learning for Recommender Systems},
  author={Cheng et al. (2016)},
  year={2016},
  note={arXiv:1606.07792}
}

• arXiv: 1606.07792