Goemotions Transfer Eval

goemotions-transfer-eval

GoEmotions: A Dataset of Fine-Grained Emotions — Demszky et al. (2020) (arXiv:2005.00547, 2020)

What this evaluates

Evaluates cross-domain generalization of emotion classification models by measuring how well a model fine-tuned on GoEmotions transfers to external benchmarks with limited labeled data, compared to training from scratch.

Datasets

• ISEAR — total 8000; splits: train (-1), test (-1)
• EmoInt — total 7000; splits: train (-1), test (-1)
• Emotion-Stimulus — total 2400; splits: train (-1), test (-1)

Metrics

• average F1-score (primary) — range: [0, 1]
  • Unweighted mean of per-class F1-scores. Per-class F1 = 2 * (Precision * Recall) / (Precision + Recall), where Precision = TP/(TP+FP) and Recall = TP/(TP+FN).

Input / output format

Input: Text instances (comments, sentences, or tweets) from target domain datasets, each associated with a single emotion label from a specific taxonomy (GoEmotions 27-class, sentiment 4-class, or Ekman 6-class).

Output: Predicted single emotion label per instance.

Scoring recipe

def compute_macro_f1(predictions, gold):
    classes = sorted(set(predictions) | set(gold))
    f1_scores = []
    for cls in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(predictions, gold) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(predictions, gold) if p != cls and g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• EmoInt uses intensity scores converted to binary labels via a 0.5 cutoff, which is non-standard and may not reflect original annotation intent.
• Training sizes vary (100, 200, 500, 1000, 80%), and results are averaged over 10 random splits per size without specifying seeds, hindering exact replication.
• The 'max' setting uses 80% of the dataset for training and 20% for testing, deviating from standard full-dataset evaluation.

Evidence (verbatim from paper)

Figure 3: Transfer learning results in terms of average F1-scores across emotion categories. ... We compare three different finetuning setups. In the BASELINE setup, we finetune BERT only on the target dataset. In the FREEZE setup, we first finetune BERT on GoEmotions, then perform transfer learning by replacing the final dense layer, freezing all layers besides the last layer and finetuning on the target dataset. The NOFREEZE setup is the same as FREEZE, except that we do not freeze the bottom layers. We hold the batch size at 16, learning rate at 2e-5 and number of epochs at 3 for all experiments.

Citation

@misc{demszky2020goemotions,
  title={GoEmotions: A Dataset of Fine-Grained Emotions},
  author={Demszky et al. (2020)},
  year={2020},
  note={arXiv:2005.00547}
}

• arXiv: 2005.00547