winoBias-eval

winoBias-eval

Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods — Zhao et al. (2018) (arXiv:1804.06876, 2018)

What this evaluates

Evaluates gender bias in coreference resolution systems by measuring performance disparity between pro-stereotypical and anti-stereotypical sentences. It probes whether models rely on gender stereotypes when resolving coreferences in challenging, Winograd-style contexts.

Datasets

• WinoBias — total ?; splits: dev (-1), test (-1)

Metrics

• F1 (primary) — range: [0, 100]
  • Standard coreference F1 score computed using scorer v8.01 (Pradhan et al., 2014).
• |Diff| — range: [0, 100]
  • Absolute difference between F1 scores on pro-stereotyped and anti-stereotyped subsets.

Input / output format

Input: Winograd-style sentences containing occupation-based entities with gendered pronouns, presented in pro-stereotypical and anti-stereotypical conditions.

Output: Set of coreference links (pairs of entity mentions) for each input sentence.

Scoring recipe

pred_links = model.predict(sentence)
gold_links = get_gold_coref(sentence)
f1 = scorer_v801.compute_f1(pred_links, gold_links)

f1_pro = compute_f1(pro_stereotypical_sentences)
f1_anti = compute_f1(anti_stereotypical_sentences)
diff = abs(f1_pro - f1_anti)

Common pitfalls

• Using a coreference scorer other than v8.01 (Pradhan et al., 2014), which yields different F1 values.
• Ignoring statistical significance testing for |Diff|; the paper requires an approximate randomized test (p < .05) to flag significant bias.
• Evaluating on standard coreference benchmarks alone, which masks bias that only appears in WinoBias's adversarial pro/anti-stereotypical split.

Evidence (verbatim from paper)

Table 2: F1 on OntoNotes and WinoBias development set. WinoBias results are split between Type-1 and Type-2 and in pro/anti-stereotypical conditions.

Citation

@misc{zhao2018genderbias,
  title={Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods},
  author={Zhao et al. (2018)},
  year={2018},
  note={arXiv:1804.06876}
}

• arXiv: 1804.06876