Trec2022 Fair Ranking Eval

trec2022-fair-ranking-eval

Overview of the TREC 2022 Fair Ranking Track — Ekstrand et al. (2023) (arXiv:2302.05558, 2023)

What this evaluates

Evaluates retrieval systems on balancing topical relevance with intersectional fairness in Wikipedia article rankings. It probes static single-query ranking for coordinators and dynamic multi-query ranking for editors under fairness constraints across demographic attributes.

Datasets

• TREC 2022 Fair Ranking Track — total ?; splits: test (-1); repo https://github.com/fair-trec/trec2022-fair-public

Metrics

• nDCG — range: [0, 1]
  • Normalized Discounted Cumulative Gain at a fixed cutoff, measuring ranking quality based on graded relevance labels.
• AWRF — range: [0, 1]
  • Average Weighted Rank Fairness, quantifying the disparity in expected exposure across protected demographic attributes.
• M1 (primary) — range: [0, 1]
  • Official Task 1 composite metric that balances nDCG and AWRF to evaluate the relevance-fairness tradeoff for static rankings.
• EE-L (primary) — range: other
  • Expected Exposure - Long term, measures long-term document exposure across a sequence of rankings. Lower values indicate better fairness.
• EE-D — range: other
  • Expected Exposure - Disparity, measures the variance or inequality in exposure across protected groups.
• EE-R — range: [0, 1]
  • Expected Exposure - Relevance, measures the relevance-weighted exposure of documents.

Input / output format

Input: Query string and a collection of Wikipedia articles. Task 1 provides 500 articles per query; Task 2 provides a pool of candidate rankings for 100 queries.

Output: Task 1: A single ranked list of 500 documents. Task 2: A sequence of 100 ranked lists, each containing 20 documents.

Scoring recipe

def score_task1(predictions, gold, protected_attrs):
    ndcg = compute_ndcg(predictions, gold)
    awrf = compute_awrf(predictions, protected_attrs)
    m1 = composite(ndcg, awrf)  # Official formula combines both
    return {'nDCG': ndcg, 'AWRF': awrf, 'M1': m1}

def score_task2(predictions_seq, gold, protected_attrs):
    ee_r = compute_expected_exposure_relevance(predictions_seq, gold)
    ee_d = compute_expected_exposure_disparity(predictions_seq, protected_attrs)
    ee_l = compute_expected_exposure_longterm(predictions_seq)
    return {'EE-R': ee_r, 'EE-D': ee_d, 'EE-L': ee_l}

Common pitfalls

• EE-L and EE-D are minimized (lower is better), whereas nDCG and EE-R are maximized.
• Fairness must be evaluated intersectionally across multiple demographic attributes (age, gender, etc.), not just single attributes.
• Task 1 evaluates static rankings per query, while Task 2 evaluates dynamic exposure across a sequence of 100 queries.

Evidence (verbatim from paper)

Table 1 shows the submitted systems ranked by the official Task 1 metric M1 and its component parts nDCG and AWRF. ... Table 4 shows the submitted systems ranked by the official Task 2 metric EE-L and its component parts EE-D and EE-R.

Citation

@misc{ekstrand2023trec2022fairranking,
  title={Overview of the TREC 2022 Fair Ranking Track},
  author={Ekstrand et al. (2023)},
  year={2023},
  note={arXiv:2302.05558}
}

• arXiv: 2302.05558