Criteo Attribution Bidding Eval

criteo-attribution-bidding-eval

Attribution Modeling Increases Efficiency of Bidding in Display Advertising — Diemert et al. (2017) (arXiv:1707.06409, 2017)

What this evaluates

Evaluates the efficiency of display advertising bidding strategies by measuring how well they align advertiser payments with actual conversion attribution. It probes the model's ability to predict conversion probability and adjust bids dynamically to avoid overbidding after early clicks, ultimately maximizing advertiser utility under budget constraints.

Datasets

• Criteo Attribution Dataset — total ?; splits: train (-1), test (-1); repo http://research.criteo.com

Metrics

• U_A (primary) — range: other
  • Attribution-aware Expected Utility (AEU) variant using the proposed exponential decay attribution model. It computes expected utility by integrating conversion probability, bid value, and cost, adjusted by a perturbation parameter β to simulate competing bid uncertainty.
• U_A* — range: other
  • Un-normed variant of the Attribution-aware Expected Utility. Used to assess robustness under high bid uncertainty, where performance gaps between bidders widen compared to the normed version.
• OEC — range: other
  • Overall Evaluation Criterion, a long-term platform metric defined as OEC ∝ (AdvertiserValue - Cost). Captures net value generated for advertisers after accounting for spend, used in online A/B testing.

Input / output format

Input: Sparse binary feature vector (via hashing trick) encoding ad impression attributes: timestamp, price paid, categorical user/ad/publisher features, click position, conversion status/value, and attribution status.

Output: Conversion probability prediction, calibrated to match a target average, which is then fed into a bidding strategy (AB, LCB, or FCB) to compute a dynamic bid value.

Scoring recipe

def evaluate_offline(train_days, test_days, beta):
    model = L2_Penalized_Logistic_Regression()
    model.fit(train_days)
    preds = model.predict(test_days)
    preds = calibrate_to_target_avg(preds)  # Ensures comparable spend
    utilities = []
    for bidder in [AB, LCB, FCB]:
        bids = bidder.compute_bid(preds, beta)
        utility = compute_expected_utility(bids, conversions, cost, beta)
        utilities.append(utility)
    return utilities

def evaluate_online(control_bids, test_bids, advertiser_value, cost):
    oec_control = advertiser_value - cost
    oec_test = advertiser_value - cost
    return (oec_test - oec_control) / oec_control  # Uplift %

Common pitfalls

• Confusing the normed utility (U_A) with the un-normed variant (U_A*), which yield different performance rankings under high bid uncertainty (low β).
• Assuming offline utility improvements directly translate to higher short-term platform revenue; the paper explicitly notes that higher efficiency reduces advertiser spend, causing a negative short-term revenue impact.
• Ignoring the calibration step: predictions must be calibrated to predict the same average value across bidders to ensure fair comparison, mimicking real-world budget feedback loops.

Evidence (verbatim from paper)

The evaluation metrics are variants of the Attribution-aware Expected Utility: $U_{A}$, $U_{A}*$, and $U_{LC}$ as presented in Section [4.2]. In order to have comparable values between bidders for a given metric, predictions are calibrated so as to predict the same value on average.

Citation

@misc{diemert2017attribution,
  title={Attribution Modeling Increases Efficiency of Bidding in Display Advertising},
  author={Diemert et al. (2017)},
  year={2017},
  note={arXiv:1707.06409}
}

• arXiv: 1707.06409