Mtop Eval

mtop-eval

MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark — Li et al. (2020) (arXiv:2008.09335, 2020)

What this evaluates

Evaluates multilingual task-oriented semantic parsing models on extracting hierarchical intent-slot representations from natural language utterances across multiple languages and transfer settings. It tests the model's ability to generalize across languages using in-language, multilingual, and zero-shot training protocols.

Datasets

• MTOP — total ?; splits: train (-1), val (-1), test (-1)
• Multilingual ATIS — total ?; splits: train (-1), val (-1), test (-1)
• Multilingual TOP — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• exact match (primary) — range: percent
  • A prediction is counted as correct only if the entire predicted intent-slot representation exactly matches the gold representation in structure and tokens. This is a strict exact-match evaluation common in semantic parsing benchmarks.

Input / output format

Input: Natural language utterance in a target language (or English for zero-shot settings).

Output: Hierarchical intent-slot representation for MTOP, or flat representation for Multilingual ATIS and Multilingual TOP.

Scoring recipe

def exact_match(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Alignment models for slot projection are imperfect, especially for low-resource languages, introducing noise and bias into zero-shot training data.
• Multilingual training requires explicit sampling ratios per language to prevent high-resource languages from dominating the batch distribution and degrading performance on low-resource languages.

Evidence (verbatim from paper)

reaching 67.2% exact match across 5 languages without target-language data. We conduct thorough experiments on the new dataset we describe in in §3. To further demonstrate the effectiveness of our proposed approaches, we also run additional experiments on the existing multilingual task-oriented semantic parsing datasets including Multilingual ATIS (Upadhyay et al., 2018) and Multilingual TOP (Schuster et al., 2019a).

Citation

@misc{li2020mtop,
  title={MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark},
  author={Li et al. (2020)},
  year={2020},
  note={arXiv:2008.09335}
}

• arXiv: 2008.09335