Erc Eval

erc-eval

SELF-EMO: Emotional Self-Evolution from Recognition to Consistent Expression — Shaowei Zhang et al. (arXiv:2604.18003, 2026)

What this evaluates

Evaluates a model's ability to recognize and classify emotional states from conversational dialogue context. It probes multi-turn emotional understanding, speaker-aware reasoning, and generalization across diverse domain settings and speaker demographics.

Datasets

• IEMOCAP — total 7433; splits: train (5163), dev (647), test (1623)
• MELD — total 13708; splits: train (9989), dev (1109), test (2610)
• EmoryNLP — total 9489; splits: train (7551), dev (954), test (984)

Metrics

• Accuracy (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly predicted emotion labels out of the total number of test instances.
• Weighted F1 (W-F1) — range: percent
  • Weighted F1 score: the harmonic mean of precision and recall per class, averaged across all classes weighted by their support (number of true instances per class) to account for label imbalance.

Input / output format

Input: Multi-turn dialogue context including previous utterances and speaker identifiers, with the model tasked to predict the emotion of the current target utterance.

Output: A single predicted emotion label corresponding to the target utterance (e.g., neutral, happy, sad, angry, or dataset-specific categories).

Scoring recipe

def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds) * 100
    classes = set(golds)
    w_f1 = 0.0
    total = len(golds)
    for cls in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == cls and g == cls)
        fp = sum(1 for p, g in zip(preds, golds) if p == cls and g != cls)
        fn = sum(1 for p, g in zip(preds, golds) if p != cls and g == cls)
        support = sum(1 for g in golds if g == cls)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        w_f1 += f1 * (support / total)
    return acc, w_f1 * 100

Common pitfalls

• Evaluating on the development split instead of the official test split, which violates the paper's strict evaluation protocol.
• Training and testing on each dataset separately (dataset-specific setting) rather than the unified multi-dataset setting reported in the main results, leading to unfair or non-comparable baselines.
• Ignoring class imbalance when reporting metrics, as ERC datasets are heavily skewed; unweighted accuracy can be misleading without Weighted F1.

Evidence (verbatim from paper)

All model evaluations are strictly performed on the official test splits of these benchmarks. ... Specifically, our model realizes an accuracy increase of +6.33% on the Qwen3-4B backbone and +8.54% on the Qwen3-8B backbone, with consistent improvements observed across all benchmarks.

Citation

@misc{zhang2026selfemo,
  title={SELF-EMO: Emotional Self-Evolution from Recognition to Consistent Expression},
  author={Shaowei Zhang et al.},
  year={2026},
  note={arXiv:2604.18003}
}

• arXiv: 2604.18003