Fair Weak Supervision Eval

fair-weak-supervision-eval

Mitigating Source Bias for Fairer Weak Supervision — Changho Shin et al. (arXiv:2303.17713, 2023)

What this evaluates

Evaluates a weak supervision pipeline's ability to mitigate labeling function bias and improve fairness across demographic groups. It measures how well a source bias mitigation method recovers accurate pseudolabels while reducing disparities in prediction rates between privileged and underrepresented groups.

Datasets

• Adult — total ?; splits: train (-1), test (-1)
• Bank Marketing — total ?; splits: train (-1), test (-1)
• CivilComments — total ?; splits: train (-1), test (-1)
• HateXplain — total ?; splits: train (-1), test (-1)
• CelebA — total ?; splits: train (-1), test (-1)
• UTKFace — total ?; splits: train (-1), test (-1)
• WRENCH — total ?; splits: train (-1), test (-1)

Metrics

• accuracy — range: [0, 1]
  • Fraction of correctly predicted labels out of total test instances.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for the positive class.
• demographic parity gap ($\Delta_{DP}$) (primary) — range: [0, 1]
  • Absolute difference in positive prediction rates between the privileged and underrepresented groups: $|P(\hat{Y}=1|G=0) - P(\hat{Y}=1|G=1)|$.
• equal opportunity gap ($\Delta_{EO}$) — range: [0, 1]
  • Absolute difference in true positive rates between groups: $|P(\hat{Y}=1|Y=1, G=0) - P(\hat{Y}=1|Y=1, G=1)|$.

Input / output format

Input: Training data with true labels (for LF evaluation) or unlabeled data, along with labeling functions (heuristics or pretrained models) that generate weak labels. Group membership labels are provided for fairness evaluation.

Output: Predicted labels (pseudolabels) from the label model, followed by final predictions from the end model (logistic regression) on the test set.

Scoring recipe

predictions = end_model.predict(test_data)
true_labels = test_labels
groups = test_groups
acc = mean(predictions == true_labels)
f1 = f1_score(true_labels, predictions)
dp_gap = abs(mean(predictions[groups==0]) - mean(predictions[groups==1]))
eo_gap = abs(mean(predictions[true_labels==1 & groups==0]) - mean(predictions[true_labels==1 & groups==1]))
return acc, f1, dp_gap, eo_gap

Common pitfalls

• Fully supervised (FS) results are provided only as an upper bound, not as a baseline for comparison.
• One-hot encoded features can distort nearest-neighbor distance calculations, causing method failure unless mitigated (e.g., via LIFT embedding).
• Highly imbalanced class distributions (e.g., P(Y=1) ≈ 0.1) can cause F1 score drops when fairness metrics are optimized.

Evidence (verbatim from paper)

To see if our method can improve both fairness and performance, we measured the demographic parity gap $(\Delta_{DP})$ and the equal opportunity gap $(\Delta_{EO})$ as fairness metrics, and computed accuracy and F1 score as performance metrics as well.

Citation

@misc{shin2023mitigating,
  title={Mitigating Source Bias for Fairer Weak Supervision},
  author={Changho Shin et al.},
  year={2023},
  note={arXiv:2303.17713}
}

• arXiv: 2303.17713