Seller Outcome Fairness Eval

seller-outcome-fairness-eval

Seller-side Outcome Fairness in Online Marketplaces — Ye et al. (2023) (arXiv:2312.03253, 2023)

What this evaluates

Evaluates the trade-off between platform revenue (GMV) and seller-side exposure fairness in online marketplace recommendation systems using simulated online environments trained on historical interaction data.

Datasets

• Proprietary Dataset — total 600000000; splits: simulation (-1)
• Electronics Event History (EVS) Dataset — total 490399; splits: simulation (-1)

Metrics

• GMV relative change (primary) — range: percent
  • (GMV_algorithm - GMV_benchmark) / GMV_benchmark * 100%. Measures the percentage change in Gross Merchandise Value (revenue) relative to a baseline algorithm without fairness regularization.
• Seller fairness target achievement — range: other
  • Count of sellers/brands where accumulated clicks/transactions over T sessions meet or exceed a predefined threshold (5 clicks per T visits for proprietary data; 5 transactions per 200,000 visits for EVS data).

Input / output format

Input: Simulated user sessions containing customer features, anchor item, and candidate recommendation items, generated by a DNN-based simulator trained on historical click/purchase data.

Output: A ranked or selected set of recommended items per user session.

Scoring recipe

gmv_alg = sum(revenue for item in alg_recommendations)
gmv_bench = sum(revenue for item in bench_recommendations)
gmv_rel = (gmv_alg - gmv_bench) / gmv_bench * 100

seller_clicks = {s: sum(clicks for s in T_sessions) for s in sellers}
target = 5 # clicks per T visits
fairness_count = sum(1 for c in seller_clicks.values() if c >= target)

Common pitfalls

• Evaluation relies on a DNN-based simulator trained on historical data, which may not capture real-world distribution shifts or user behavior dynamics.
• Proprietary dataset details (exact size, feature distribution, and target threshold T) are masked due to confidentiality, preventing exact replication.
• Fairness is measured at the seller/brand level rather than the individual item level, fundamentally changing the allocation objective.

Evidence (verbatim from paper)

Notice that, in the table, we report the relative percentage change in outcomes lifted by Algorithm 1 w.r.t. benchmark algorithms. Table 1 shows that with the increasing value of the β parameter, we observe that the number of sellers reaching the target outcomes significantly increases for larger β for both tests.

Citation

@misc{ye2023seller,
  title={Seller-side Outcome Fairness in Online Marketplaces},
  author={Ye et al. (2023)},
  year={2023},
  note={arXiv:2312.03253}
}

• arXiv: 2312.03253