Mcwq Eval

mcwq-eval

On Evaluating Multilingual Compositional Generalization with Translated Datasets — Wang et al. (2023) (arXiv:2306.11420, 2023)

What this evaluates

Evaluates a model's ability to generalize compositionally to unseen syntactic structures and cross-lingual settings in semantic parsing. It measures how well models translate natural language questions into correct SPARQL queries across monolingual and zero-shot cross-lingual scenarios.

Datasets

• MCWQ — total ?; splits: MCD_mean (-1), Random (-1)

Metrics

• Exact Match (%) (primary) — range: percent
  • Percentage of test instances where the model's predicted SPARQL query exactly matches the ground truth query character-for-character.

Input / output format

Input: Natural language question in English, Japanese, or Chinese.

Output: A SPARQL query or logical form representing the semantic parsing of the question.

Scoring recipe

correct = 0
total = len(predictions)
for pred, gold in zip(predictions, golds):
    if pred.strip() == gold.strip():
        correct += 1
exact_match_acc = (correct / total) * 100

Common pitfalls

• Ground truth (GT) translated datasets contain noise and semantic distortions from neural machine translation, which can unfairly penalize models compared to rule-based (RBMT) translations.
• Adding auxiliary objectives (e.g., question reconstruction and language prediction) to cross-lingual parsers can dramatically decrease performance on compositional generalization tasks compared to logical-form-only baselines.

Evidence (verbatim from paper)

Table 3: Monolingual experiment results: Exact match accuracies in percentage (%) are shown here. We present the model performance on the two translated datasets, which share the English branch. MCD_mean represents the average accuracy across 3 MCD splits, and the detailed results breakdown can be found in Appendix D.1. Random refers to the results on the random split.

Citation

@misc{wang2023multilingual,
  title={On Evaluating Multilingual Compositional Generalization with Translated Datasets},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2306.11420}
}

• arXiv: 2306.11420