Fairness Sequential Eval

fairness-sequential-eval

Fairness under uncertainty in sequential decisions — Lee et al. (2026) (arXiv:2604.21711, 2026)

What this evaluates

Evaluates sequential decision policies under simulated historical and measurement bias to measure how accounting for unrealized outcomes affects fairness disparities and cumulative utility. It probes whether uncertainty-aware exploration mitigates selection rate differences and false positive rate parity violations without sacrificing profit.

Datasets

• Synthetic Sequential Simulation — total ?; splits: simulation (-1)

Metrics

• selection rate difference (primary) — range: percent
  • Difference in the proportion of candidates from the minority group (A=0) versus the majority group who are given a loan.
• FPR parity — range: percent
  • Difference in false positive rates between sensitive groups, computed over observed outcomes.
• cumulative profit — range: other
  • Total utility/profit accumulated over the sequential time steps (Q1-Q10).

Input / output format

Input: Sequential decision instances over 10 quarters, including binary sensitive attribute A, observed outcomes Y (subject to measurement/historical bias), and policy actions.

Output: Policy decisions/actions per quarter, with final evaluation computed as aggregate fairness metrics and cumulative profit over the horizon.

Scoring recipe

def score(actions, sensitive_attrs, outcomes, horizon=10):
    sel_minority = sum(actions[sensitive_attrs == 0]) / len(sensitive_attrs[sensitive_attrs == 0])
    sel_majority = sum(actions[sensitive_attrs == 1]) / len(sensitive_attrs[sensitive_attrs == 1])
    selection_rate_diff = abs(sel_minority - sel_majority)
    
    fpr_minority = fpr(actions[sensitive_attrs == 0], outcomes[sensitive_attrs == 0])
    fpr_majority = fpr(actions[sensitive_attrs == 1], outcomes[sensitive_attrs == 1])
    fpr_parity = abs(fpr_minority - fpr_majority)
    
    cumulative_profit = sum(outcome_values for t in range(horizon))
    return {'selection_rate_diff': selection_rate_diff, 'fpr_parity': fpr_parity, 'cumulative_profit': cumulative_profit}

Common pitfalls

• Simulations are synthetic and explicitly designed to illustrate mechanisms rather than benchmark deployable fairness algorithms.
• FPR parity results are noted as noisy due to the small scale of the simulation.
• Profit scale is narrow (spanning only 6 to 7), making initial disparities appear larger than they are.

Evidence (verbatim from paper)

Selection rate refers to the proportion of candidates from a specific group who are given a loan, with selection rate difference then being the difference in rate selection between the two sensitive groups. Other fairness metrics are more standard, and defined in [appendix B] Evaluation metrics ‣ Fairness under uncertainty in sequential decisions. Fig. 3 compares only the final values of the metrics at the end of our given time period (Q1-Q10).

Citation

@misc{lee2026fairness,
  title={Fairness under uncertainty in sequential decisions},
  author={Lee et al. (2026)},
  year={2026},
  note={arXiv:2604.21711}
}

• arXiv: 2604.21711