Culemo Eval

culemo-eval

CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding — Belay et al. (2025) (arXiv:2503.10688, 2025)

What this evaluates

Probes LLMs' cross-cultural emotion understanding by testing their ability to predict emotions and sentiments across six languages, specifically examining how prompt language and explicit country context influence model performance.

Datasets

• CULEMO — total ?; splits: test (-1); repo https://github.com/llm-for-emotion/culemo

Metrics

• emotion prediction (primary) — range: [0, 1]
  • Accuracy of predicted emotion categories compared to gold annotations, evaluated via zero-shot text generation.
• sentiment analysis — range: [0, 1]
  • Accuracy of predicted sentiment polarity compared to gold annotations, evaluated via zero-shot text generation.

Input / output format

Input: Instruction, input text, and expected answer format in either English or one of five target languages (Arabic, Amharic, German, Hindi, Spanish), optionally prefixed with explicit country context (e.g., 'You live in <>,').

Output: Generated text containing the predicted emotion or sentiment label, extracted using the PEDANTS tool.

Scoring recipe

def compute_metric(predictions, gold):
    # Extract predicted label from model output using PEDANTS tool
    pred_labels = [extract_label(pred) for pred in predictions]
    # Calculate accuracy
    correct = sum(1 for p, g in zip(pred_labels, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Using English prompts for non-English cultures significantly underperforms compared to in-language prompting with country context.
• Relying on translated datasets introduces cultural bias; native cultural annotations are required for valid evaluation.
• Zero-shot text generation requires robust label extraction; without tools like PEDANTS, parsing free-form model outputs is error-prone.

Evidence (verbatim from paper)

Specifically, we explore culture-aware emotion understanding via two main tasks: (1) emotion prediction and (2) sentiment analysis. All tested models are instruction-fine-tuned, except for the Aya-expanse model. We also experiment with prompts that do and do not include explicit country context, using the phrase "You live in <>," (where <> is one of the six targeted countries: UAE, USA, Ethiopia, Germany, India, and Mexico). Each task is framed as a text-generation problem, and the models are evaluated in a zero-shot setting.

Citation

@misc{belay2025culemo,
  title={CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understanding},
  author={Belay et al. (2025)},
  year={2025},
  note={arXiv:2503.10688}
}

• arXiv: 2503.10688