Massive Eval

massive-eval

MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages — FitzGerald et al. (2022) (arXiv:2204.08582, 2022)

What this evaluates

Evaluates multilingual natural language understanding capabilities, specifically intent classification and slot filling, across 51 typologically diverse languages. It measures model robustness to different scripts, spacing conventions, and zero-shot cross-lingual transfer scenarios.

Datasets

• MASSIVE — total 1000000; splits: train (-1), val (-1), test (-1); repo https://github.com/alexa/massive

Metrics

• intent accuracy — range: percent
  • Percentage of utterances where the predicted intent label exactly matches the gold intent label.
• microaveraged slot F1 score — range: percent
  • Micro-averaged F1 score computed across all token-level slot labels (including 'Other') for the slot filling task.
• exact match accuracy (primary) — range: percent
  • Percentage of utterances where both the predicted intent and the entire sequence of slot labels exactly match the gold annotations.

Input / output format

Input: Raw text utterance. For mT5 Text-to-Text models, the input is prefixed with 'Annotate:' followed by the utterance.

Output: Intent classification: a single intent label. Slot filling: a sequence of slot labels (including 'Other') for each token, followed by the intent label.

Scoring recipe

def compute_metrics(preds, gold):
    intent_correct = sum(1 for p, g in zip(preds, gold) if p[0] == g[0])
    intent_acc = intent_correct / len(preds)
    exact_match = sum(1 for p, g in zip(preds, gold) if p == g)
    exact_match_acc = exact_match / len(preds)
    tp = fp = fn = 0
    for p_slots, g_slots in zip([p[1] for p in preds], [g[1] for g in gold]):
        for t in set(p_slots + g_slots):
            tp += sum(1 for a, b in zip(p_slots, g_slots) if a == t and b == t)
            fp += sum(1 for a, b in zip(p_slots, g_slots) if a == t and b != t)
            fn += sum(1 for a, b in zip(p_slots, g_slots) if a != t and b == t)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    slot_f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return intent_acc, slot_f1, exact_match_acc

Common pitfalls

• Artificial character-level spacing for non-space-delimited languages (e.g., Japanese, Chinese) disrupts pretrained token embeddings, causing severe performance drops that may not reflect true model capability.
• Zero-shot evaluation trains only on English (en-US) and tests on all non-English locales, which differs from standard multilingual training and can be confused with full-dataset cross-lingual transfer.
• Microaveraged slot F1 and exact match accuracy are highly sensitive to tokenization and spacing conventions, making cross-lingual comparisons difficult without consistent preprocessing.

Evidence (verbatim from paper)

We trained our models with the Adam optimizer (Kingma and Ba, 2017) and chose the best performing model checkpoint based on overall exact match accuracy across all locales. Table 3 shows the results for each model and training setup, including those for the best performing locale, the worst performing locale, and locale-averaged results for intent accuracy, microaveraged slot F1 score, and exact match accuracy.

Citation

@misc{fitzgerald2022massive,
  title={MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages},
  author={FitzGerald et al. (2022)},
  year={2022},
  note={arXiv:2204.08582}
}

• arXiv: 2204.08582