Counterfactual Fairness QA Eval

counterfactual-fairness-qa-eval

Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System — Mayilvaghanan et al. (2026) (arXiv:2602.14970, 2026)

What this evaluates

Evaluates whether LLM-based contact center QA systems exhibit systematic bias when agent identity (gender, ethnicity, religion, disability) or contextual factors (past performance, behavioral style) are counterfactually altered. Measures if model judgments change disproportionately based on these attributes rather than transcript content.

Datasets

• Contact-Center QA Transcripts — total 4200; splits: test (4200)

Metrics

• Counterfactual Flip Rate (CFR) (primary) — range: percent
  • Proportion of counterfactual pairs where the model's binary QA judgment flips (e.g., from 'Yes' to 'No') when protected attributes or contextual cues are altered.
• Mean Absolute Score Difference (MASD) — range: score units
  • Average absolute difference in model output scores across counterfactual attribute manipulations.

Input / output format

Input: Contact-center transcripts (real or synthetic) paired with 30 distinct Auto-QA questions. Transcripts are counterfactually modified to alter agent identity, past performance, or behavioral style.

Output: Binary QA judgments ('Yes'/'No') and evaluation scores generated by the LLMs for each transcript-question pair.

Scoring recipe

# For each counterfactual pair (original, modified):
pred_orig = model.predict(transcript_orig, qa_question)
pred_mod = model.predict(transcript_mod, qa_question)
# CFR: proportion of pairs where prediction flips
flips = sum(1 for p1, p2 in zip(preds_orig, preds_mod) if p1 != p2)
CFR = flips / total_pairs
# MASD: average absolute difference in scores
MASD = mean(abs(score_orig - score_mod) for score_orig, score_mod in scores_pairs)

Common pitfalls

• Confounding genuine fairness violations with model stochasticity without establishing a robustness baseline.
• Relying solely on automated filtering for counterfactual generation without human verification for semantic equivalence.
• Assuming that reducing fairness disparities will not degrade task accuracy; requires joint interpretation of fairness and accuracy metrics.

Evidence (verbatim from paper)

Our experiment evaluates the fairness of 18 prominent LLMs using a corpus of 3000 real contact-center transcripts and 8 LLMs on 1200 synthetic transcripts generated using ConvoGen. The study employs 30 distinct Auto-QA questions, with the dataset balanced across questions and their binary answers (“Yes” and “No”) to avoid skewed distributions. Results show systematic disparities (CFR from 5.4% to 16.4%), with contextual priming and implicit identity cues causing severe bias, and fairness not correlated with accuracy.

Citation

@misc{mayilvaghanan2026counterfactual,
  title={Counterfactual Fairness Evaluation of LLM-Based Contact Center Agent Quality Assurance System},
  author={Mayilvaghanan et al. (2026)},
  year={2026},
  note={arXiv:2602.14970}
}

• arXiv: 2602.14970