Unpie Eval

unpie-eval

Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you! — Chung et al. (2024) (arXiv:2410.01023, 2024)

What this evaluates

Assesses multimodal language models' ability to resolve lexical ambiguity in puns using visual context. It probes visual-textual alignment, multimodal literacy, and the capacity to disambiguate or reconstruct ambiguous text when provided with explanatory or disambiguating images.

Datasets

• UNPIE — total 1000; splits: test (1000)

Metrics

• exact-match accuracy (primary) — range: [0, 1]
  • Measures the percentage of generated pun phrases that exactly match the ground-truth target phrase after case-insensitive string comparison.
• BERTScore — range: [0, 1]
  • Text similarity metric that computes cosine similarity between contextual embeddings of predicted and reference texts, averaged across precision, recall, and F1.
• Win Rate (%) — range: percent
  • Proportion of cases where the similarity score between the disambiguated translation and the reference exceeds the similarity score between an unconditional baseline translation and the reference.
• Cohen Kappa ($\kappa$) — range: [0, 1]
  • Statistical measure of inter-annotator agreement used to validate GPT-4's meaning frequency rankings against human-annotated ground truth.

Input / output format

Input: English pun sentence paired with either a single explanation image (depicting both meanings) or two disambiguator images (each depicting one meaning). For translation tasks, the input may also include target language instructions.

Output: For pun grounding: the exact target pun phrase. For translation/reconstruction: the translated sentence in the target language (French, German, or Korean). For disambiguation: a binary classification label indicating which meaning/image is correct.

Scoring recipe

def exact_match_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return correct / len(gold)

def bertscore(predictions, gold):
    # Uses pre-trained BERT embeddings to compute similarity
    scores = compute_bertscore(predictions, gold)
    return np.mean(scores)

def win_rate(disambiguated, unconditional, references):
    s1 = [bertscore(u, r) for u, r in zip(unconditional, references)]
    s2 = [bertscore(d, r) for d, r in zip(disambiguated, references)]
    wins = sum(1 for s2_val, s1_val in zip(s2, s1) if s2_val > s1_val)
    return wins / len(references)

Common pitfalls

• Images used for disambiguation must not contain explicit text or watermarks, as the benchmark explicitly filters these out to avoid rewarding OCR rather than visual understanding.
• Translation targets are intentionally disambiguated; literal translations that preserve the original pun's ambiguity are considered incorrect for the reconstruction task.
• The dataset splits puns into homographic and heterographic categories, and performance should be reported separately for each rather than only as an aggregate.

Evidence (verbatim from paper)

We report the exact match accuracy of the generated pun phrase. Text similarity was evaluated using BERTScore. We measure text similarity scores for each pair: $s_{1}=sim(\hat{y}{0},\hat{y}{1})$ and $s_{2}=sim(\hat{y}{0},y)$, and compute the win rate as the proportion of cases where $s{2}$ exceeds $s_{1}$.

Citation

@misc{chung2024unpie,
  title={Can visual language models resolve textual ambiguity with visual cues? Let visual puns tell you!},
  author={Chung et al. (2024)},
  year={2024},
  note={arXiv:2410.01023}
}

• arXiv: 2410.01023