Faircontrast Tabular Eval

faircontrast-tabular-eval

FairContrast: Enhancing Fairness through Contrastive learning and Customized Augmenting Methods on Tabular Data — Tayebi et al. (2025) (arXiv:2510.02017, 2025)

What this evaluates

This evaluation probes a model's ability to learn fair representations from tabular data by balancing predictive accuracy with demographic parity. It measures how effectively the model mitigates bias across privileged and unprivileged groups defined by sensitive attributes such as gender or age.

Datasets

• Adult — total 48842; splits: unspecified (-1)
• German Credit — total 1000; splits: unspecified (-1)
• Heritage Health — total 50000; splits: unspecified (-1)

Metrics

• Accuracy — range: [0, 1]
  • Fraction of correctly predicted labels out of total instances.
• Demographic Parity (DP) (primary) — range: [0, 1]
  • Absolute difference in the rate of positive predictions between the privileged group (S=1) and the unprivileged group (S=0). Lower values indicate higher fairness.

Input / output format

Input: Tabular feature vectors containing standard attributes and a sensitive attribute column (e.g., gender or age), along with a target label.

Output: Predicted class label (or probability) for each instance.

Scoring recipe

def compute_metrics(y_true, y_pred, sensitive_attr):
    accuracy = np.mean(y_pred == y_true)
    p_priv = np.mean(y_pred[sensitive_attr == 1])
    p_unpriv = np.mean(y_pred[sensitive_attr == 0])
    dp = abs(p_priv - p_unpriv)
    return accuracy, dp

Common pitfalls

• Results are reported as average accuracy and maximum DP over five random seeds, not a single run.
• DP is mathematically defined for binary sensitive attributes, but Heritage Health uses nine age categories; the paper does not specify how DP is aggregated for multi-category sensitive attributes.
• The privileged group (S=1) definition varies by dataset (e.g., older age for German Credit, specific gender for Adult) and must be correctly assigned before computing DP.

Evidence (verbatim from paper)

In this study, we adopt Demographic Parity (DP), also known as statistical parity, as our main fairness metric. Demographic parity ensures that the probability of receiving a favorable outcome is being equitably distributed across groups(privileged and unprivileged). Specifically, this metric requires that the likelihood of all positive predictions (both true positives and false positives) be similar across these groups. Thus, discrimination or disparities can be quantified by measuring the difference between the conditional probabilities of positive predictions for the privileged and unprivileged groups: |P(Ŷ=1∣X,S=1)−P(Ŷ=1∣X,S=0)|

Citation

@misc{tayebi2025faircontrast,
  title={FairContrast: Enhancing Fairness through Contrastive learning and Customized Augmenting Methods on Tabular Data},
  author={Tayebi et al. (2025)},
  year={2025},
  note={arXiv:2510.02017}
}

• arXiv: 2510.02017