Humorrank Eval

humorrank-eval

HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models — Ajayi et al. (2026) (arXiv:2604.19786, 2026)

What this evaluates

Evaluates LLM humor generation by conducting pairwise preference judgments on joke outputs across 300 diverse prompts. It measures how well models master comedic mechanisms rather than relying on model scale, using an LLM-as-judge framework to produce global rankings.

Datasets

• SemEval-2026 MWAHAHA — total 300; splits: test (300)

Metrics

• Bradley-Terry Maximum Likelihood Estimation (primary) — range: [0, 1]
  • Pairwise preferences are aggregated using Bradley-Terry Maximum Likelihood Estimation to derive global win probabilities and rankings. Stable Elo is also used as a sequence-agnostic validation metric.

Input / output format

Input: A prompt condition and two joke outputs generated by different models.

Output: A binary preference label indicating which of the two jokes is funnier, accompanied by structured reasoning based on the GTVH taxonomy.

Scoring recipe

# Collect pairwise preferences across all model pairs and prompts
# W[i, j] = count where model i preferred over j
# N[i, j] = total comparisons between i and j
# Solve Bradley-Terry MLE for skill parameters theta:
# max_theta sum_{i<j} [ W[i,j]*log(exp(theta[i])/(exp(theta[i])+exp(theta[j]))) + W[j,i]*log(exp(theta[j])/(exp(theta[i])+exp(theta[j]))) ]
# Compute win probability P(i beats j) = exp(theta[i]) / (exp(theta[i]) + exp(theta[j]))
# Rank models by descending theta[i] or average win probability

Common pitfalls

• LLM-as-judge bias: The primary judge may exhibit systematic preferences, requiring cross-judge ablation and human validation.
• Prompt-specific variance: Humor quality varies significantly across the 300 conditions; aggregating across all prompts is necessary to avoid cherry-picking.
• Scale vs. Mechanism: Larger models do not automatically score higher; mastery of comedic mechanisms is the primary driver of high rankings.

Evidence (verbatim from paper)

It employs an Adaptive Swiss Tournament to efficiently generate comparisons across nine models on the SemEval-2026 MWAHAHA dataset, followed by Bradley-Terry Maximum Likelihood Estimation to produce statistically robust, globally consistent rankings.

Citation

@misc{ajayi2026humorrank,
  title={HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models},
  author={Ajayi et al. (2026)},
  year={2026},
  note={arXiv:2604.19786}
}

• arXiv: 2604.19786