Nlu Service Eval

nlu-service-eval

Benchmarking Natural Language Understanding Services for building Conversational Agents — Liu et al. (2019) (arXiv:1903.05566, 2019)

What this evaluates

Evaluates commercial and open-source NLU platforms on intent classification and named entity recognition across multiple dialogue domains, highlighting limitations in multi-intent support and contextual modeling.

Datasets

• NLU Evaluation Dataset — total 11036; splits: train (-1), test (-1); repo https://github.com/xliuhw/NLU-Evaluation-Data

Metrics

• Intent classification accuracy, Entity recognition precision (primary) — range: percent
  • Standard classification metrics: accuracy for intent classification, and precision/recall/F1 for entity recognition. The paper specifically highlights precision differences for entity recognition.

Input / output format

Input: Raw user utterance text.

Output: Predicted intent label(s) and extracted entity spans with types for each utterance.

Scoring recipe

# 10-fold cross-validation (90% train, 10% test per fold)
for fold in range(10):
    train_set = dataset[fold * 0.9 : (fold + 1) * 0.9]
    test_set = dataset[(fold + 1) * 0.9 : (fold + 1)]
    predictions = call_nlu_api(test_set)
    intent_acc = sum(p.intent == g.intent for p, g in zip(predictions, test_set)) / len(test_set)
    entity_prec = compute_entity_precision(predictions, test_set)
    # Aggregate metrics across folds

Common pitfalls

• Dataset is inherently unbalanced across intents and entities (e.g., some intents have as few as 77 instances), which may skew per-class metrics.
• API versioning and configuration differences across platforms (e.g., Dialogflow V1.0 vs V2.0, Watson version matching) can significantly impact results.
• Platforms lack support for multiple intents per utterance and dialogue context, limiting joint modeling evaluation.

Evidence (verbatim from paper)

For the evaluation experiments we report below, we performed 10 fold cross-validation with 90% of the subcorpus for training and 10% for testing in each fold. ... Watson leads in Intent classification but lags in Entity Recognition due to low precision

Citation

@misc{liu2019nlu,
  title={Benchmarking Natural Language Understanding Services for building Conversational Agents},
  author={Liu et al. (2019)},
  year={2019},
  note={arXiv:1903.05566}
}

• arXiv: 1903.05566