qhjqhj00 intent-detection-eval
Efficient Intent Detection with Dual Sentence Encoders — Casanueva et al. (2020) (arXiv:2003.04807, 2020)
What this evaluates
This benchmark evaluates the ability of pretrained sentence encoders and classifiers to correctly identify user intents from conversational utterances. It specifically probes few-shot generalization by testing models on severely limited training data (10 or 30 examples per intent) while maintaining a standard full test set.
Datasets
- BANKING77 — total ?; splits: train (-1), test (-1)
- CLINC150 — total ?; splits: train (-1), test (-1)
- HWU64 — total ?; splits: train (-1), test (-1)
Metrics
accuracy(primary) — range: percent- The proportion of correctly predicted intent labels out of the total number of test instances. Reported as a percentage (×100%).
Input / output format
Input: A single user utterance (text string), encoded via a pretrained sentence encoder.
Output: A predicted intent class label corresponding to one of the predefined intent categories.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return (correct / len(gold_labels)) * 100
Common pitfalls
- Few-shot training sets contain only 10 or 30 examples per intent, not 10 or 30 total across all intents.
- The test set is never subsampled; it remains the full standard test set for every experimental run.
- BERT-FIXED uses mean-pooled sequence output for encoding, whereas BERT-TUNED fine-tunes the full model and uses the [CLS] token output.
Evidence (verbatim from paper)
We report accuracy as the main evaluation measure for all experimental runs. We experiment with setups where only 10 or 30 examples are available for each intent, while we use the same standard test sets for each experimental run.
Citation
@misc{casanueva2020efficient,
title={Efficient Intent Detection with Dual Sentence Encoders},
author={Casanueva et al. (2020)},
year={2020},
note={arXiv:2003.04807}
}
- arXiv: 2003.04807