Gm Annotation Eval

gm-annotation-eval

Toward Automatic Group Membership Annotation for Group Fairness Evaluation — Chen et al. (2024) (arXiv:2407.08926, 2024)

What this evaluates

Tests the ability of language models to automatically classify documents into predefined group membership categories (e.g., gender, geographic location) for group fairness evaluation in information retrieval.

Datasets

• TREC fair ranking track 2021 — total ?; splits: train (500), test (100); repo https://github.com/fm-chen/nldb-experiments
• TREC fair ranking track 2022 — total ?; splits: train (500), test (100); repo https://github.com/fm-chen/nldb-experiments
• NTCIR fairweb1 (Chuweb-21D) — total ?; splits: train (500), test (100); repo https://github.com/fm-chen/nldb-experiments

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted group membership labels out of the total number of test documents.

Input / output format

Input: Cleaned full-text field of a document (special characters removed, stop words removed, lemmatized, truncated to 512 tokens).

Output: A single categorical label from a predefined set of group membership subgroups (e.g., 'male', 'female', 'non-binary', or specific geographic locations).

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
return correct / total if total > 0 else 0.0

Common pitfalls

• Human annotation for group membership is highly costly, time-consuming, and suffers from severe class imbalance (e.g., many 'unknown' labels).
• Generative LLM performance is highly sensitive to prompt design and pre-training data distribution, unlike discriminative models.
• Aggregated fairness metrics can mask document-level annotation errors, potentially hiding poor individual classification performance.

Evidence (verbatim from paper)

outperforming generative large language models (LLMs) like GPT and Mistral in accuracy with minimal supervision.

Citation

@misc{chen2024toward,
  title={Toward Automatic Group Membership Annotation for Group Fairness Evaluation},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2407.08926}
}

• arXiv: 2407.08926