Multilingual Intent Classification Eval

multilingual-intent-classification-eval

From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service — He et al. (2026) (arXiv:2603.23172, 2026)

What this evaluates

Evaluates multilingual intent classification capabilities in logistics customer service, measuring how well models route user queries to parent or leaf intent categories across seen and unseen languages. It specifically probes the performance gap between native and machine-translated queries to reveal how synthetic translation overestimates model robustness in real-world routing scenarios.

Datasets

• Logistics Customer Service Intent Benchmark — total 30000; splits: train (-1), dev (-1), test (-1)

Metrics

• Accuracy/Micro-F1 (primary) — range: [0, 1]
  • For single-label classification, Accuracy and Micro-F1 are identical: the total number of correctly predicted intents divided by the total number of instances. It reflects overall routing accuracy under the benchmark's naturally imbalanced traffic distribution.
• Macro-F1 — range: [0, 1]
  • The unweighted mean of the F1 scores computed per class. Reported in the appendix to analyze long-tail performance and class balance independently of majority-class dominance.

Input / output format

Input: A single user query text in one of the benchmark languages (English, Spanish, Arabic, Indonesian, or Chinese). For closed-source LLMs, queries are provided within few-shot prompting templates.

Output: A single intent label selected from a fixed inventory of either 13 parent intents or 17 leaf intents, depending on the evaluated task setting.

Scoring recipe

def compute_accuracy_micro_f1(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    total = len(gold_labels)
    accuracy = correct / total
    # For single-label classification, Micro-F1 equals Accuracy
    micro_f1 = accuracy
    return micro_f1

Common pitfalls

• Models are evaluated under the benchmark's natural traffic distribution without class rebalancing, so Accuracy/Micro-F1 heavily favors majority intents and masks long-tail failures.
• Evaluating on machine-translated test queries consistently overestimates model robustness compared to native queries, particularly for long-tail intents and cross-lingual transfer.
• The hierarchical classification task uses a deployment-oriented taxonomy where some parent nodes map to a single leaf, which can artificially inflate hierarchy-aware decoding accuracy if not handled carefully.

Evidence (verbatim from paper)

Our primary metric is Accuracy/Micro-F1. This choice is intentional: the benchmark preserves the naturally imbalanced traffic distribution of real customer-service systems, and overall routing accuracy is the most deployment-relevant objective in this setting. We report Macro-F1 in Appendix[C] for long-tail and class-balance analysis.

Citation

@misc{he2026synthetic,
  title={From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service},
  author={He et al. (2026)},
  year={2026},
  note={arXiv:2603.23172}
}

• arXiv: 2603.23172