Ner Gender Bias Eval

ner-gender-bias-eval

Man is to Person as Woman is to Location: Measuring Gender Bias in Named Entity Recognition — Ninareh Mehrabi et al. (2019) (arXiv:1910.10872, 2019)

What this evaluates

This benchmark probes gender bias and temporal drift in Named Entity Recognition (NER) systems. It measures whether models disproportionately misclassify female names compared to male names, and how this bias shifts across 139 years of U.S. census data. The evaluation specifically tests statistical parity in entity recognition under varying contextual templates.

Datasets

• U.S. Census Names (1880-2018) — total ?; splits: test (-1); repo https://github.com/Ninarehm/NERGenderBias

Metrics

• error rate (primary) — range: [0, 1]
  • The proportion of names incorrectly classified as non-PERSON entities out of the total names evaluated. It is calculated as 1 minus recognition accuracy.

Input / output format

Input: A single sentence constructed from one of nine templates, starting with a historical U.S. census name followed by a human-like activity phrase.

Output: A sequence of entity tags for each token in the sentence, specifically indicating whether the name token is labeled as PERSON or another entity type (e.g., LOCATION).

Scoring recipe

def compute_error_rate(predictions, gold_tags):
    total = len(gold_tags)
    if total == 0:
        return 0.0
    errors = sum(1 for p, g in zip(predictions, gold_tags) if p != g)
    return errors / total

Common pitfalls

• Ignoring template variation: The benchmark uses 9 different sentence templates to control for context, so evaluating only on raw names misses contextual bias.
• Overlooking temporal stratification: Bias must be analyzed across the 139-year span (1880-2018), not just as an aggregate, as model version and historical name distributions shift over time.

Evidence (verbatim from paper)

We test these models against 139 years of U.S. census data from years 1880 to 2018. Our benchmark evaluates these models based upon how well they recognize these names as a PERSON entity. Template 1, containing just the name, purely tests the name itself and reveals something about the distribution of the training data. Template 4 is designed to direct the model to tag the name as a person. Template 3 may reveal more subtle gender bias that stems from society as, historically, men have received greater levels of education from women. It is possible that the error rate may be even higher for female names under Template 3.

Citation

@misc{mehrabi2019nergenderbias,
  title={Man is to Person as Woman is to Location: Measuring Gender Bias in Named Entity Recognition},
  author={Ninareh Mehrabi et al. (2019)},
  year={2019},
  note={arXiv:1910.10872}
}

• arXiv: 1910.10872