qhjqhj00 mdgender-eval
Multi-Dimensional Gender Bias Classification — Dinan et al. (2020) (arXiv:2005.00614, 2020)
What this evaluates
Evaluates models' ability to classify pragmatic gender bias in text across three conversational dimensions: ABOUT (topic-related), AS (speaker role/attitude), and TO (addressee-directed). It probes fine-grained detection of gendered language and contextual bias cues.
Datasets
- MDGENDER — total ?; splits: test (-1)
Metrics
percentage accuracy(primary) — range: percent- Calculated as the number of correctly classified instances divided by the total number of evaluated instances, multiplied by 100. Only the masculine, feminine, and neutral classes are included; the unknown class is excluded.
Input / output format
Input: Text utterances or sentences annotated for gender bias along the ABOUT, AS, and TO dimensions.
Output: A single class label: 'masculine', 'feminine', or 'neutral'.
Scoring recipe
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
if gold in ['masculine', 'feminine', 'neutral']:
total += 1
if pred == gold:
correct += 1
accuracy = (correct / total) * 100 if total > 0 else 0
Common pitfalls
- Excluding the 'unknown' class from evaluation, as the paper explicitly states it is not modeled and should be dropped.
- Assuming single-task classifiers generalize across dimensions; the paper shows they specialize to one dimension and fail to transfer.
- Confusing surface-form cues (e.g., pronouns) with contextual bias; ABOUT relies on surface forms while TO/AS require deeper context understanding.
Evidence (verbatim from paper)
We measure the percentage accuracy for masculine, feminine, and neutral classes. We do not evaluate on the unknown class, as it is not modeled. Classifier results on MDGENDER are shown in Table 3.
Citation
@misc{dinan2020multidimensional,
title={Multi-Dimensional Gender Bias Classification},
author={Dinan et al. (2020)},
year={2020},
note={arXiv:2005.00614}
}
- arXiv: 2005.00614