Ethioemo Eval

ethioemo-eval

Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding — Belay et al. (2024) (arXiv:2412.17837, 2024)

What this evaluates

Evaluates LLMs' ability to classify multiple emotions in low-resource Ethiopian languages (Amharic, Afan Oromo, Somali, Tigrinya) and English. It probes cross-lingual transfer capabilities, the effectiveness of zero-shot and few-shot prompting strategies, and the impact of fine-tuning on multi-label emotion understanding tasks.

Datasets

• EthioEmo — total ?; splits: test (-1); repo https://github.com/Tadesse-Destaw/EthioEmo

Metrics

• Weighted-averaged F1-score (primary) — range: percent
  • Label-wise F1 score averaged with weights proportional to the number of true instances per label (support). Reported as a percentage.

Input / output format

Input: Text input in one of the target languages (Amharic, Afan Oromo, Somali, Tigrinya, or English), optionally accompanied by k-shot examples and a system prompt.

Output: A set of emotion labels predicted for the input text (multi-label classification).

Scoring recipe

def weighted_f1_multilabel(y_true, y_pred, num_labels):
    f1_scores = []
    support = []
    for l in range(num_labels):
        tp = sum(1 for yt, yp in zip(y_true, y_pred) if yt[l] == 1 and yp[l] == 1)
        fp = sum(1 for yt, yp in zip(y_true, y_pred) if yt[l] == 0 and yp[l] == 1)
        fn = sum(1 for yt, yp in zip(y_true, y_pred) if yt[l] == 1 and yp[l] == 0)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
        support.append(sum(yt[l] for yt in y_true))
    total_support = sum(support)
    return sum(f * s for f, s in zip(f1_scores, support)) / total_support * 100

Common pitfalls

• Performance varies significantly across languages due to pre-training data inclusion.
• Translation to English does not fully bridge the performance gap, indicating cultural/linguistic nuances in emotion expression.
• Prompt sensitivity is high; results are averaged over three different prompt templates to mitigate this.

Evidence (verbatim from paper)

Table 4: Weighted-averaged F1-score results from fine-tuned pre-trained language models.

Citation

@misc{belay2024ethioemo,
  title={Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding},
  author={Belay et al. (2024)},
  year={2024},
  note={arXiv:2412.17837}
}

• arXiv: 2412.17837