Eq Bench Eval

eq-bench-eval

EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models — Paech (2023) (arXiv:2312.06281, 2023)

What this evaluates

Evaluates large language models' ability to understand and rate emotional intensity in conflict-driven dialogue scenarios. It probes emotional intelligence through automated scoring of model-generated ratings, avoiding subjective human interpretation.

Datasets

• EQ-Bench — total ?; splits: test (-1); repo https://github.com/EQ-bench/EQ-Bench

Metrics

• EQ-Bench Score (primary) — range: [0, 100]
  • Average of automated emotional intensity ratings across test instances, normalized to a 0–100 scale. Models must produce at least 50 parseable answers to achieve a passing score.

Input / output format

Input: Conflict-driven dialogue scenarios (generated by GPT-4) presented via a prompt that instructs the model to rate emotional intensity and includes a critique & revision section.

Output: Numerical emotional intensity rating, optionally followed by a critique and revised rating if the multi-step prompt is used. Must be machine-parseable.

Scoring recipe

def compute_eq_bench_score(predictions, gold):
    valid = [(p, g) for p, g in zip(predictions, gold) if is_parseable(p)]
    if len(valid) < 50:
        return 0.0  # Fails threshold
    raw_scores = [score_intensity(p, g) for p, g in valid]
    return normalize_to_100(mean(raw_scores))

Common pitfalls

• Models must yield at least 50 parseable answers; failure results in a FAIL or near-zero score.
• Omitting the critique & revision step in the prompt significantly lowers scores (by ~9.3% on average) and deviates from the official protocol.
• Cross-benchmark comparisons require normalizing scores to a 0–100 scale, which can mask raw distribution differences.

Evidence (verbatim from paper)

OpenAI's GPT-4-0613 model produced the highest EQ-Bench score by a considerable margin. All models listed in Table 1 produced enough parseable answers (at least 50) for a passing score, with the exception of OpenAI's older generation Curie, ADA and Babbage models.

Citation

@misc{paech2023eqbench,
  title={EQ-Bench: An Emotional Intelligence Benchmark for Large Language Models},
  author={Paech (2023)},
  year={2023},
  note={arXiv:2312.06281}
}

• arXiv: 2312.06281