Fig QA Eval

fig-qa-eval

Testing the Ability of Language Models to Interpret Figurative Language — Liu et al. (2022) (arXiv:2204.12632, 2022)

What this evaluates

Evaluates language models' ability to interpret nonliteral, creative metaphors by selecting the correct literal meaning from two opposing options, and generating sensible interpretations for novel metaphors. It probes commonsense grounding and contextual understanding beyond literal paraphrase tasks.

Datasets

• Fig-QA — total ?; splits: train (-1), test (-1); repo https://github.com/nightingal3/Fig-QA

Metrics

• accuracy (primary) — range: [0, 1]
  • Percentage of correct predictions out of total test instances. Evaluated for both inference (selecting between two interpretations) and generation (manual labeling of completions as correct/incorrect/literal).

Input / output format

Input: A figurative metaphor phrase paired with two opposing literal interpretations (inference), or a metaphor phrase alone (generation).

Output: Selection of one of the two interpretations (inference), or a generated text completion (generation).

Scoring recipe

# Inference
pred = model.predict(metaphor, [interp1, interp2])
correct = 1 if pred == gold_label else 0
accuracy = sum(correct) / len(test_set)

# Generation
completions = model.generate(metaphor, max_tokens=100)
labels = [annotator.label(c) for c in completions]
correct = sum(1 for l in labels if l == 'correct')
accuracy = correct / len(non_ambiguous_instances)

Common pitfalls

• Backward direction (matching a literal phrase to a metaphor) is significantly harder than the forward direction.
• Paired/group scoring scheme drastically penalizes autoregressive models compared to standard accuracy.
• Generation evaluation requires manual annotation and is highly sensitive to temperature settings and truncation rules.

Evidence (verbatim from paper)

The first question is whether strong LMs can interpret metaphors at all when presented with two opposing meanings, in zero-shot or supervised settings. ... GPT-3 Davinci's accuracy, counting literalized metaphors as incorrect, was 50.8%. Not counting literalized metaphors, the accuracy was 63.9%.

Citation

@misc{liu2022figqa,
  title={Testing the Ability of Language Models to Interpret Figurative Language},
  author={Liu et al. (2022)},
  year={2022},
  note={arXiv:2204.12632}
}

• arXiv: 2204.12632