Trec2020 Fairness Ranking Eval

trec2020-fairness-ranking-eval

University of Washington at TREC 2020 Fairness Ranking Track — Feng et al. (2020) (arXiv:2011.02066, 2020)

What this evaluates

Evaluates information retrieval and re-ranking systems on their ability to balance document relevance with demographic fairness. It probes how well algorithms maintain ranking utility while ensuring equitable representation across inferred gender and country groups.

Datasets

• TREC 2020 Fairness Ranking Track dataset — total ?; splits: test (-1)

Metrics

• utility (primary) — range: [0, 1]
  • Measures the relevance quality of the ranked list, typically computed via standard IR metrics like NDCG or MAP. The paper notes it drops significantly when relevance scores are excluded from the ranking objective.
• fairness — range: [0, 1]
  • Quantifies demographic parity or group diversity in the ranked results. Calculated based on the distribution of inferred gender and country attributes among top-ranked documents.

Input / output format

Input: Query, a pool of candidate documents, and inferred demographic attributes (gender and country) for each document's author.

Output: A ranked list of documents.

Scoring recipe

# Pseudo-code for evaluation
relevance_scores = compute_bm25(query, documents)
group_attrs = infer_demographics(documents) # via genderize.io/geolocation
ranked_docs = re_rank(relevance_scores, group_attrs, weights=[w_r, w_g, w_c])
utility = compute_ndcg(ranked_docs, relevance_scores)
fairness = compute_group_diversity(ranked_docs, group_attrs)
return utility, fairness

Common pitfalls

• Group definitions rely on external APIs (genderize.io, geolocation) which may produce noisy or missing attributes, affecting fairness calculations.
• Random ranking baselines show highly variable fairness depending on group definitions, making them unreliable for direct comparison.
• Optimizing for fairness often causes a sharp drop in relevance utility, requiring careful weight tuning to find a viable trade-off.

Evidence (verbatim from paper)

In both gender and country groups, BM25 demonstrates a relatively high utility score but a low fairness score, implying that BM25 fails to take fairness into account when calculating the ranking.

Citation

@misc{feng2020fairnessranking,
  title={University of Washington at TREC 2020 Fairness Ranking Track},
  author={Feng et al. (2020)},
  year={2020},
  note={arXiv:2011.02066}
}

• arXiv: 2011.02066