# Meta4xnlies Eval

> Evaluates multilingual models' ability to detect metaphorical expressions at the token level and interpret them within a Natural Language Inference (NLI) framework across English and Spanish. It probes cross-lingual transfer, domain generalization, and the impact of metaphorical content on model reasoning. Use when the user wants to benchmark on Meta4XNLI, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/meta4xnlies-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/meta4xnlies-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/meta4xnlies-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/meta4xnlies-eval

---


# meta4xnlies-eval

> Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation — Sanchez-Bayona et al. (2024) (arXiv:2404.07053, 2024)

## What this evaluates

Evaluates multilingual models' ability to detect metaphorical expressions at the token level and interpret them within a Natural Language Inference (NLI) framework across English and Spanish. It probes cross-lingual transfer, domain generalization, and the impact of metaphorical content on model reasoning.

## Datasets

- **Meta4XNLI** — total ?; splits: train (-1), development (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted labels (entailment/neutral/contradiction for NLI, or metaphor/non-metaphor for detection) out of total instances.

## Input / output format

**Input**: Premise and hypothesis sentence pairs (for NLI/interpretation) or single sentences/token sequences (for detection), provided in English or Spanish.

**Output**: For detection: token-level labels (metaphor/non-metaphor). For interpretation: one of three NLI relations [entailment, natural, contradiction].

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)
```

## Common pitfalls

- The paper uses 'natural' instead of the standard 'neutral' for the NLI label, which may cause parsing issues if automated scripts expect standard NLI labels.
- Evaluation is split by metaphor presence (pairs with metaphors vs. without), requiring careful stratification to avoid performance masking.
- Cross-domain and zero-shot cross-lingual setups require strict separation of source datasets to prevent data leakage.

## Evidence (verbatim from paper)

> Models were prompted to answer with one of the three NLI relations [entailment, natural, contradiction]. To do so, we designed two prompts available in Appendix Table [29]: one with no examples (zero-shot) and another one with longer context and one example for each label (chain-of-thought (CoT)).

## Citation

```bibtex
@misc{sanchezbayona2024meta4xnlies,
  title={Meta4XNLI: A Crosslingual Parallel Corpus for Metaphor Detection and Interpretation},
  author={Sanchez-Bayona et al. (2024)},
  year={2024},
  note={arXiv:2404.07053}
}
```

- arXiv: 2404.07053