Idiomaticity Detection Eval

idiomaticity-detection-eval

Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection — Phelps et al. (2024) (arXiv:2405.09279, 2024)

What this evaluates

Evaluates large language models' ability to disambiguate whether a given phrase is used idiomatically or literally within a specific context. It probes zero-shot, few-shot, and cross-lingual prompting capabilities, measuring how well models generalize to idiomatic expressions without task-specific fine-tuning.

Datasets

• SemEval 2022 Task 2a — total ?; splits: test (916)
• FLUTE — total ?; splits: test (-1)
• MAGPIE — total ?; splits: test (-1)

Metrics

• macro F1 (primary) — range: [0, 1]
  • Macro F1 score computed across all classes (idiomatic vs. literal) and languages, averaging the F1 per class without regard to class distribution.

Input / output format

Input: For SemEval/MAGPIE: 'Expression: . Context: .' For FLUTE: 'Sentence 1: Sentence 2: .' Few-shot variants prepend available training examples per phrase.

Output: A single character: 'i' for idiomatic or 'l' for literal. For FLUTE: 'entailment' or 'contradiction'.

Scoring recipe

def compute_macro_f1(predictions, gold):
    valid = [(p, g) for p, g in zip(predictions, gold) if p in ('i', 'l')]
    if not valid: return 0.0
    preds, golds = zip(*valid)
    y_true = [1 if g == 'i' else 0 for g in golds]
    y_pred = [1 if p == 'i' else 0 for p in preds]
    f1_0 = f1_score(y_true, y_pred, pos_label=0)
    f1_1 = f1_score(y_true, y_pred, pos_label=1)
    return (f1_0 + f1_1) / 2

Common pitfalls

• LLMs frequently violate strict output constraints, returning extra text instead of just 'i' or 'l'; the protocol treats these as invalid and penalizes them.
• High run-to-run variance (up to 0.04 F1) means single-run results are unreliable; the authors recommend averaging 2-3 runs.
• Few-shot prompting can cause models to memorize dataset artifacts (e.g., predicting the same label for a given PIE) rather than learning the task, particularly for smaller models.

Evidence (verbatim from paper)

“Disambiguate whether the given expression is used idiomatically or literally in the given context, returning ’i’ if the expression is being used idiomatically or ’l’ if literally. Expression: . Context: . Only return one letter (i or l).” Table 4: Results (macro F1) on the English test set of SemEval with GPT-3.5-turbo using prompt engineering.

Citation

@misc{phelps2024sign,
  title={Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection},
  author={Phelps et al. (2024)},
  year={2024},
  note={arXiv:2405.09279}
}

• arXiv: 2405.09279