Semeval 2022 Task2 Eval

semeval-2022-task2-eval

SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding — Madabushi et al. (2022) (SemEval-2022, 2022)

What this evaluates

Evaluates language models' ability to detect whether a multi-word expression (MWE) in a sentence is used idiomatically or literally, and to model the semantic similarity of idiomatic expressions. It tests compositionality understanding and contextual semantic representation across English, Portuguese, and Galician.

Datasets

• SemEval-2022 Task 2 — total ?; splits: train (-1), test (-1); repo https://github.com/H-TayyarMadabushi/SemEval_2022_Task2-idiomaticity

Metrics

• macro F1 score (primary) — range: [0, 1]
  • The unweighted mean of recall or precision computed per class (idiomatic vs. literal), then averaged across classes. Commonly used for imbalanced binary classification.
• Spearman correlation — range: [-1, 1]
  • Rank-based correlation coefficient measuring the monotonic relationship between predicted similarity scores and gold human similarity ratings. Computed as 1 - (6 * sum(d_i^2)) / (n * (n^2 - 1)) for ranks.

Input / output format

Input: A sentence containing a target multi-word expression (MWE) to be evaluated for idiomaticity (Subtask A) or a pair of sentences/expressions for semantic similarity (Subtask B).

Output: Subtask A: Binary label (idiomatic or literal). Subtask B: Continuous similarity score.

Scoring recipe

def compute_macro_f1(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Subtask A uses an imbalanced dataset, requiring careful handling of class distribution (e.g., focal loss or data augmentation).
• Idiomatic expressions lack compositional meaning, causing standard pre-trained embeddings to perform poorly without fine-tuning or specialized masking strategies.
• Evaluation is conducted across three languages (English, Portuguese, Galician), and performance can vary significantly across low-resource vs. high-resource languages.

Evidence (verbatim from paper)

The evaluation metric is macro F1 score, and the ranking is based on the 'All' column.

Citation

@misc{madabushi2022semeval,
  title={SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding},
  author={Madabushi et al. (2022)},
  year={2022},
  note={SemEval-2022}
}

• arXiv: 2204.10050