Fairness Explanation Eval

fairness-explanation-eval

Counterfactual Explanation for Fairness in Recommendation — Xiangmeng Wang et al. (2023) (arXiv:2307.04386, 2023)

What this evaluates

This benchmark evaluates the faithfulness and utility of counterfactual explanations in recommendation systems. It measures how effectively generated explanations identify fairness-disparaging attributes by iteratively erasing them and observing the resulting impact on recommendation accuracy and item exposure inequality.

Datasets

• Yelp — total 198397; splits: train (-1), val (-1), test (-1)
• Douban Movie — total 1068278; splits: train (-1), val (-1), test (-1)
• Last-FM — total 92834; splits: train (-1), val (-1), test (-1)

Metrics

• NDCG@K (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff K. Measures the quality of the ranked recommendation list by weighting relevance scores by their logarithmic position.
• Hit Ratio (HR)@K — range: [0, 1]
  • Binary metric indicating whether at least one relevant item appears in the top-K recommendation list.
• Head-tailed Rate (HT)@K — range: [0, 1]
  • Ratio of head-tailed (popular) item count to the list length K. Lower values indicate reduced popularity bias.
• Gini@K — range: [0, 1]
  • Gini coefficient measuring inequality between head-tailed and long-tailed items within the top-K recommendation list. Lower values indicate fairer exposure distribution.

Input / output format

Input: Historical user-item interaction logs, user attributes, and item attributes from heterogeneous information networks, processed into fused user and item embeddings.

Output: Top-K ranked item lists and a set of attribute-level counterfactual explanations (specific user/item attributes to remove for fairness diagnostics).

Scoring recipe

def evaluate_erasure(model, explanations, K, E, ground_truth):
    # explanations: list of attribute sets per user
    # E: number of top attributes to erase from explanation
    erased_attrs = explanations[:E]
    # Remove erased attributes from attribute space
    new_emb = fuse_embeddings(model.user_emb, model.item_emb, exclude=erased_attrs)
    # Generate Top-K recommendations
    recs = generate_topk(new_emb, K)
    # Compute metrics
    ndcg = calc_ndcg(recs, ground_truth, K)
    hr = calc_hr(recs, ground_truth, K)
    ht = calc_head_tailed_rate(recs, K)
    gini = calc_gini_coefficient(recs, K)
    return ndcg, hr, ht, gini

Common pitfalls

• HT@K and Gini@K are inverse fairness metrics: lower values indicate better fairness, opposite to NDCG@K and HR@K.
• The evaluation requires a chronological split and a 10-core filter; skipping these preprocessing steps invalidates dataset density and temporal fairness baselines.
• E (erasure length) and K (ranking cutoff) are distinct hyperparameters; results must be reported for both E∈{5,10,20} and K∈{5,20,40}.

Evidence (verbatim from paper)

Given the recommendation results at each evaluation point, we use Normalized Discounted Cumulative Gain (NDCG)@K and Hit Ratio (HR)@K to measure the recommendation performance. As this work focuses on item exposure fairness in recommendations, we use two wildly-adopted item-side evaluation metrics, i.e., Head-tailed Rate (HT)@K and Gini@K, for fairness evaluation.

Citation

@misc{wang2023counterfactual,
  title={Counterfactual Explanation for Fairness in Recommendation},
  author={Xiangmeng Wang et al. (2023)},
  year={2023},
  note={arXiv:2307.04386}
}

• arXiv: 2307.04386