# Massive Eval

> Evaluates multilingual natural language understanding capabilities, specifically intent classification and slot filling, across 51 typologically diverse languages. It measures model robustness to different scripts, spacing conventions, and zero-shot cross-lingual transfer scenarios. Use when the user wants to benchmark on MASSIVE, or asks about evaluating this task. Reports exact match accuracy.

- Skill: `qhjqhj00/massive-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/massive-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/massive-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/massive-eval

---


# massive-eval

> MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages — FitzGerald et al. (2022) (arXiv:2204.08582, 2022)

## What this evaluates

Evaluates multilingual natural language understanding capabilities, specifically intent classification and slot filling, across 51 typologically diverse languages. It measures model robustness to different scripts, spacing conventions, and zero-shot cross-lingual transfer scenarios.

## Datasets

- **MASSIVE** — total 1000000; splits: train (-1), val (-1), test (-1); repo https://github.com/alexa/massive

## Metrics

- `intent accuracy` — range: percent
  - Percentage of utterances where the predicted intent label exactly matches the gold intent label.
- `microaveraged slot F1 score` — range: percent
  - Micro-averaged F1 score computed across all token-level slot labels (including 'Other') for the slot filling task.
- `exact match accuracy` **(primary)** — range: percent
  - Percentage of utterances where both the predicted intent and the entire sequence of slot labels exactly match the gold annotations.

## Input / output format

**Input**: Raw text utterance. For mT5 Text-to-Text models, the input is prefixed with 'Annotate:' followed by the utterance.

**Output**: Intent classification: a single intent label. Slot filling: a sequence of slot labels (including 'Other') for each token, followed by the intent label.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    intent_correct = sum(1 for p, g in zip(preds, gold) if p[0] == g[0])
    intent_acc = intent_correct / len(preds)
    exact_match = sum(1 for p, g in zip(preds, gold) if p == g)
    exact_match_acc = exact_match / len(preds)
    tp = fp = fn = 0
    for p_slots, g_slots in zip([p[1] for p in preds], [g[1] for g in gold]):
        for t in set(p_slots + g_slots):
            tp += sum(1 for a, b in zip(p_slots, g_slots) if a == t and b == t)
            fp += sum(1 for a, b in zip(p_slots, g_slots) if a == t and b != t)
            fn += sum(1 for a, b in zip(p_slots, g_slots) if a != t and b == t)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    slot_f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return intent_acc, slot_f1, exact_match_acc
```

## Common pitfalls

- Artificial character-level spacing for non-space-delimited languages (e.g., Japanese, Chinese) disrupts pretrained token embeddings, causing severe performance drops that may not reflect true model capability.
- Zero-shot evaluation trains only on English (en-US) and tests on all non-English locales, which differs from standard multilingual training and can be confused with full-dataset cross-lingual transfer.
- Microaveraged slot F1 and exact match accuracy are highly sensitive to tokenization and spacing conventions, making cross-lingual comparisons difficult without consistent preprocessing.

## Evidence (verbatim from paper)

> We trained our models with the Adam optimizer (Kingma and Ba, 2017) and chose the best performing model checkpoint based on overall exact match accuracy across all locales. Table 3 shows the results for each model and training setup, including those for the best performing locale, the worst performing locale, and locale-averaged results for intent accuracy, microaveraged slot F1 score, and exact match accuracy.

## Citation

```bibtex
@misc{fitzgerald2022massive,
  title={MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages},
  author={FitzGerald et al. (2022)},
  year={2022},
  note={arXiv:2204.08582}
}
```

- arXiv: 2204.08582