qhjqhj00 ediref-erc-efr
SemEval 2024 -- Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF) — Shivani Kumar et al. (2024) (arXiv:2402.18944, 2024)
What this evaluates
Evaluates emotion recognition and emotion-flip reasoning in multi-party conversations, specifically identifying trigger utterances that cause emotional shifts in both code-mixed (Hindi-English) and monolingual English dialogues.
Datasets
- E-MaSaC — total ?; splits: train (-1), test (-1); repo https://github.com/LCS2-IIITD/EDiReF-SemEval2024.git
- MELD-FR — total ?; splits: train (-1), test (-1)
Metrics
weighted F1(primary) — range: [0, 1]- Macro-averaged F1 score weighted by class support (number of true instances for each class).
F1 score for trigger utterances(primary) — range: [0, 1]- Standard F1 score computed over binary classification of utterances as triggers vs non-triggers for emotion flips.
Input / output format
Input: Conversation context consisting of a sequence of utterances with speaker and emotion annotations (Task A), or conversation text for identifying the preceding utterance that causes an emotional shift (Task B/C).
Output: Task A: predicted emotion label per utterance. Task B/C: binary prediction indicating whether each utterance is the trigger for the emotion flip.
Scoring recipe
def compute_f1(y_true, y_pred):
tp = sum(1 for t, p in zip(y_true, y_pred) if t == p == 1)
fp = sum(1 for t, p in zip(y_true, y_pred) if t != 1 and p == 1)
fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p != 1)
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
Common pitfalls
- Implicit triggers: emotion flips can be caused by external events not mentioned in the dialogue, leading to no marked trigger utterances.
- Trivial baseline: the i-1 utterance is frequently the trigger, allowing rule-based or simple models to achieve high scores without deep reasoning.
- Class imbalance: the dataset contains significantly more negative and neutral emotions than positive ones, biasing models toward predicting Joy or Neutral.
Evidence (verbatim from paper)
The results for Task A are compiled in Table 4. Out of the 24 submitted papers, 22 teams explored the code-mixed ERC task, attaining weighted F1 scores spanning from 0.70 to 0.18.
Citation
@misc{kumar2024ediref,
title={SemEval 2024 -- Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF)},
author={Shivani Kumar et al. (2024)},
year={2024},
note={arXiv:2402.18944}
}
- arXiv: 2402.18944