Up5 Fairness Eval

up5-fairness-eval

UP5: Unbiased Foundation Model for Fairness-aware Recommendation — Hua et al. (2023) (arXiv:2305.12090, 2023)

What this evaluates

Evaluates recommendation accuracy and counterfactual fairness of LLM-based recommendation models. It measures ranking performance using Hit@k metrics and assesses bias by calculating the AUC for predicting sensitive user attributes from recommendations.

Datasets

• MovieLens-1M — total ?; splits: train (-1), val (-1), test (-1)
• Insurance — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Hit@1 (primary) — range: [0, 1]
  • 1 if the ground truth item is ranked within the top 1 predictions, else 0.
• Hit@3 — range: [0, 1]
  • 1 if the ground truth item is ranked within the top 3 predictions, else 0.
• Hit@10 — range: [0, 1]
  • 1 if the ground truth item is ranked within the top 10 predictions, else 0.
• AUC — range: [0, 1]
  • Area Under the ROC Curve for predicting user sensitive attributes (gender, age, occupation, marital status) from recommendations. Lower values indicate better fairness.

Input / output format

Input: User interaction history (sequence) and a candidate set containing 1 positive item and 100 randomly selected negative items.

Output: Predicted relevance scores or a ranked list of items for recommendation; predicted probabilities for sensitive attributes.

Scoring recipe

def hit_at_k(predictions, ground_truth, k):
    return 1 if ground_truth in predictions[:k] else 0

def auc_score(y_true, y_pred):
    from sklearn.metrics import roc_auc_score
    return roc_auc_score(y_true, y_pred)

Common pitfalls

• Leave-one-out split convention: the second-to-last interacted item is strictly used for validation and the last item for testing.
• Negative sampling is fixed at exactly 100 random negatives per positive sample during evaluation.
• AUC measures fairness inversely: lower AUC for sensitive attribute prediction indicates better fairness, unlike standard accuracy metrics.

Evidence (verbatim from paper)

To evaluate direct recommendation and sequential recommendation tasks, one correct item is predicted among 100 randomly selected negative samples for both tasks. The metrics are Hit@k for k in {1, 3, 10}. We adopt the commonly used leave-one-out strategy (for each user, treat the second-to-last interacted item to be the validation item and the last interacted item to be the test item) to create the training, validation, and test datasets. We adopt AUC for user attribute classification to evaluate whether sensitive attributes are involved in recommendations.

Citation

@misc{hua2023up5,
  title={UP5: Unbiased Foundation Model for Fairness-aware Recommendation},
  author={Hua et al. (2023)},
  year={2023},
  note={arXiv:2305.12090}
}

• arXiv: 2305.12090