Holisticbias Eval

holisticbias-eval

"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset — Smith et al. (2022) (arXiv:2205.09209, 2022)

What this evaluates

Evaluates demographic and intersectional biases in language models by measuring disparities in token likelihoods, generation styles, and offensiveness across a curated set of demographic descriptor terms embedded in sentence templates.

Datasets

• HOLISTICBIAS — total 600; splits: test (-1)

Metrics

• Full Gen Bias (primary) — range: other
  • Measures the disparity in generation style probabilities (e.g., sympathy, envy, curiosity) across demographic descriptor axes. Computed by prompting the model with HOLISTICBIAS templates, extracting style probabilities via a classifier, and aggregating differences or variances across descriptor groups.

Input / output format

Input: Templated dialogue sentences with demographic descriptor terms inserted (e.g., '[NOUN PHRASE] is a person.', 'I love [NOUN PHRASE]', etc.)

Output: Model-generated text responses or token probability distributions.

Scoring recipe

def compute_full_gen_bias(prompts, model, style_classifier):
    style_probs = []
    for prompt in prompts:
        response = model.generate(prompt)
        probs = style_classifier.predict(response)
        style_probs.append(probs)
    axis_groups = group_by_axis(style_probs, prompts)
    bias_score = compute_variance_or_diff(axis_groups)
    return bias_score

Common pitfalls

• Bias metrics are highly sensitive to the choice of style classifier (e.g., B.A.D. classifier) and its calibration.
• Perplexity and pseudo-log-likelihood differences can be confounded by the pre-training frequency of descriptor terms, making it hard to isolate learned bias from data exposure.

Evidence (verbatim from paper)

Full measurements of the bias in DialoGPT and BlenderBot 2.0 3B are shown in Table 8 for Full Gen Bias and Partial Gen Bias and in Table 9 for Summed-Cluster Gen Bias.

Citation

@misc{smith2022holisticbias,
  title={"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset},
  author={Smith et al. (2022)},
  year={2022},
  note={arXiv:2205.09209}
}

• arXiv: 2205.09209