Fairco Dynamic Ltr Eval

fairco-dynamic-ltr-eval

Controlling Fairness and Bias in Dynamic Learning-to-Rank — Morik et al. (2020) (arXiv:2005.14713, 2020)

What this evaluates

This evaluation probes a dynamic learning-to-rank algorithm's ability to balance ranking quality with group-level fairness under position bias. It tests whether the model can maintain high relevance-based ranking performance while actively controlling exposure and impact disparities between predefined item groups over a sequence of user interactions.

Datasets

• Ad Fontes Media Bias (semi-synthetic news) — total ?; splits: simulation (-1); repo https://github.com/MarcoMorik/Dynamic-Fairness
• MovieLens-20M — total ?; splits: simulation (-1)

Metrics

• average cumulative NDCG (primary) — range: [0, 1]
  • Average over time $\tau$ of the DCG utility $U^{DCG}(\sigma_t|\mathbf{r}_t)$ computed on the true relevance vector $\mathbf{r}_t$ and ranking $\sigma_t$. Normalized by ideal DCG.
• Exposure Unfairness — range: other
  • Group disparity metric $\overline{D}_{\tau}^{E}$ measuring differences in cumulative exposure allocation between groups over time $\tau$ (defined in Eq. 16).
• Impact Unfairness — range: other
  • Group disparity metric $\overline{D}_{\tau}^{I}$ measuring differences in cumulative impact (relevance-weighted exposure) between groups over time $\tau$ (defined in Eq. 16).

Input / output format

Input: A set of items (news articles or movies) with group labels and features, plus a user context vector (polarity/openness parameters for news, or learned embeddings for movies).

Output: A ranking permutation $\sigma_t$ of the items for the current user.

Scoring recipe

def compute_metrics(rankings, true_relevances, group_labels, tau):
    ndcg_sum = 0.0
    group_exposure = {g: 0.0 for g in set(group_labels)}
    for t in range(tau):
        # NDCG calculation
        dcg = sum(true_relevances[t][i] / log2(i + 2) for i in range(len(rankings[t])))
        idcg = sum(sorted(true_relevances[t], reverse=True)[i] / log2(i + 2) for i in range(len(rankings[t])))
        ndcg_sum += dcg / idcg if idcg > 0 else 0
        # Exposure tracking for unfairness
        for pos, item in enumerate(rankings[t]):
            exposure = 1.0 / log2(pos + 2)
            group_exposure[group_labels[item]] += exposure
    avg_ndcg = ndcg_sum / tau
    # Simplified proxy for Eq. 16 disparity
    unfairness = abs(group_exposure['left'] - group_exposure['right'])
    return avg_ndcg, unfairness

Common pitfalls

• Confusing Exposure Unfairness with Impact Unfairness; the paper shows optimizing one can increase the other (Fig. 10).
• Assuming debiased relevance estimation (IPS) alone ensures fairness; it corrects position bias but does not control group disparity without an explicit fairness controller.
• Treating the semi-synthetic news simulation as a direct substitute for real-world click logs; user behavior is strictly modeled via a Position-based Click Model (PBM) with synthetic polarities.

Evidence (verbatim from paper)

We measure ranking quality by the average cumulative NDCG $\frac{1}{\tau}\sum_{t=1}^{\tau}U^{DCG}(\sigma_t|\mathbf{r}t)$ over all the users up to time $\tau$. We measure Exposure Unfairness via $\overline{D}{\tau}^{E}$ and Impact Unfairness via $\overline{D}_{\tau}^{I}$ as defined in Equation (16).

Citation

@misc{morik2020controllingfairness,
  title={Controlling Fairness and Bias in Dynamic Learning-to-Rank},
  author={Morik et al. (2020)},
  year={2020},
  note={arXiv:2005.14713}
}

• arXiv: 2005.14713