Tid 8 Eval

tid-8-eval

You Are What You Annotate: Towards Better Models through Annotator Representations — Naihao Deng et al. (2023) (arXiv:2305.14663, 2023)

What this evaluates

Probes a model's ability to learn from inherently subjective or disagreed-upon annotations by treating each annotator's label as a separate example, rather than aggregating them into a single ground truth label.

Datasets

• TID-8 — total ?; splits: train (-1), test (-1), dev (-1); repo https://github.com/MichiganNLP/Annotator-Embeddings

Metrics

• exact match accuracy (primary) — range: [0, 1]
  • Percentage of predictions that exactly match the gold label.
• macro F1 — range: [0, 1]
  • Unweighted mean of F1 scores computed per class, treating all classes equally regardless of frequency.

Input / output format

Input: Example text, optionally concatenated with an annotator ID.

Output: A predicted label.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    em = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    classes = sorted(set(gold_labels))
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / len(f1_scores)
    return em, macro_f1

Common pitfalls

• Aggregating multiple annotator labels into a single ground truth instead of treating each annotation as a separate example.
• Using a standard data split that mixes annotators across train and test sets, violating the required 'annotation split' where both sets must contain the same annotators.

Evidence (verbatim from paper)

Evaluation Metrics. We report exact match accuracy (EM accuracy) and macro F1 scores on annotator-specific labels.

Citation

@misc{deng2023annotator,
  title={You Are What You Annotate: Towards Better Models through Annotator Representations},
  author={Naihao Deng et al. (2023)},
  year={2023},
  note={arXiv:2305.14663}
}

• arXiv: 2305.14663