qhjqhj00 new-yorker-caption-humor-eval
Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest — Hessel et al. (2022) (arXiv:2209.06293, 2022)
What this evaluates
Evaluates AI's ability to understand humor through multimodal and text-based tasks, including matching captions to cartoons, ranking caption quality, and generating humorous explanations. It probes indirect allusion, cultural context, and visual-linguistic reasoning.
Datasets
- New Yorker Caption Contest — total 704; splits: 5-fold CV (-1)
Metrics
Accuracy(primary) — range: [0, 1]- Proportion of correctly matched captions or selected winners against ground truth labels. Calculated as the mean of correct predictions over total instances.
CrowdAcc— range: [0, 1]- Accuracy against crowd-sourced quality rankings for cartoon captions. Measures how well models predict general audience preferences.
NYAcc— range: [0, 1]- Accuracy against New Yorker editor selections for cartoon captions. Measures alignment with professional editorial taste.
Pairwise Win Rate— range: [0, 1]- Percentage of instances where one model's explanation is preferred over another's in a head-to-head comparison. Determined by majority vote of three crowd workers per instance.
Input / output format
Input: For matching/ranking: cartoon image (From Pixels) or human-authored cartoon description (From Description) concatenated with multiple-choice caption options. For explanation: cartoon description and/or caption conditioned for autoregressive generation.
Output: For matching/ranking: a single letter corresponding to the selected caption choice. For explanation: a generated natural language text explaining the humor.
Scoring recipe
def compute_accuracy(predictions, gold):
return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
def compute_pairwise_win_rate(model_a_exps, model_b_exps, gold_votes):
wins_a = 0
for a, b, votes in zip(model_a_exps, model_b_exps, gold_votes):
if votes.count('A') > votes.count('B'):
wins_a += 1
return wins_a / len(gold_votes)
Common pitfalls
- Human performance is not an upper bound for matching/ranking because labels reflect varying tastes and are not from a single annotator.
- Surface-level automatic metrics like BLEU-4 and Rouge-L do not correlate with human preference for explanation quality.
- Zero-shot prompting universally incurs performance drops compared to few-shot or fine-tuned setups across all tasks.
Evidence (verbatim from paper)
Table 2 contains the results. Among the from description models, GPT-4 (5-shot) generally performs best, e.g., achieving 84.5% accuracy on matching. It (and fine-tuned GPT-3) also perform better at predicting New Yorker editor selections than our three humans (column NYAcc: GPT-3 69.8 vs. Human estimate, 64.6), but underperform at predicting crowd selections (CrowdAcc column: GPT-4 73.3 vs. 83.7).
Citation
@misc{hessel2022newyorkerhumor,
title={Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest},
author={Hessel et al. (2022)},
year={2022},
note={arXiv:2209.06293}
}
- arXiv: 2209.06293