# Mcwq Eval

> Evaluates a model's ability to generalize compositionally to unseen syntactic structures and cross-lingual settings in semantic parsing. It measures how well models translate natural language questions into correct SPARQL queries across monolingual and zero-shot cross-lingual scenarios. Use when the user wants to benchmark on MCWQ, or asks about evaluating this task. Reports Exact Match (%).

- Skill: `qhjqhj00/mcwq-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mcwq-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mcwq-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mcwq-eval

---


# mcwq-eval

> On Evaluating Multilingual Compositional Generalization with Translated Datasets — Wang et al. (2023) (arXiv:2306.11420, 2023)

## What this evaluates

Evaluates a model's ability to generalize compositionally to unseen syntactic structures and cross-lingual settings in semantic parsing. It measures how well models translate natural language questions into correct SPARQL queries across monolingual and zero-shot cross-lingual scenarios.

## Datasets

- **MCWQ** — total ?; splits: MCD_mean (-1), Random (-1)

## Metrics

- `Exact Match (%)` **(primary)** — range: percent
  - Percentage of test instances where the model's predicted SPARQL query exactly matches the ground truth query character-for-character.

## Input / output format

**Input**: Natural language question in English, Japanese, or Chinese.

**Output**: A SPARQL query or logical form representing the semantic parsing of the question.

## Scoring recipe

```python
correct = 0
total = len(predictions)
for pred, gold in zip(predictions, golds):
    if pred.strip() == gold.strip():
        correct += 1
exact_match_acc = (correct / total) * 100
```

## Common pitfalls

- Ground truth (GT) translated datasets contain noise and semantic distortions from neural machine translation, which can unfairly penalize models compared to rule-based (RBMT) translations.
- Adding auxiliary objectives (e.g., question reconstruction and language prediction) to cross-lingual parsers can dramatically decrease performance on compositional generalization tasks compared to logical-form-only baselines.

## Evidence (verbatim from paper)

> Table 3: Monolingual experiment results: Exact match accuracies in percentage (%) are shown here. We present the model performance on the two translated datasets, which share the English branch. MCD_mean represents the average accuracy across 3 MCD splits, and the detailed results breakdown can be found in Appendix D.1. Random refers to the results on the random split.

## Citation

```bibtex
@misc{wang2023multilingual,
  title={On Evaluating Multilingual Compositional Generalization with Translated Datasets},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2306.11420}
}
```

- arXiv: 2306.11420