Mixatis Mixsnips Eval

mixatis-mixsnips-eval

Co-guiding Net: Achieving Mutual Guidances between Multiple Intent Detection and Slot Filling via Heterogeneous Semantics-Label Graphs — Xing et al. (2022) (arXiv:2210.10375, 2022)

What this evaluates

Evaluates joint intent detection and slot filling on mixed-domain conversational datasets. It probes a model's ability to simultaneously predict multiple intents per utterance and extract corresponding slot entities, measuring both token-level and sentence-level alignment.

Datasets

• MixATIS — total 14746; splits: train (13162), val (756), test (828)
• MixSNIPS — total 44173; splits: train (39776), val (2198), test (2199)

Metrics

• Intent Accuracy — range: percent
  • Percentage of utterances where the predicted set of intents exactly matches the ground truth set.
• Slot F1 — range: percent
  • Standard token-level F1 score for slot labeling, computed over all tokens across the dataset.
• Overall Accuracy (primary) — range: percent
  • Ratio of sentences where both the predicted intent set and all predicted slot labels exactly match the ground truth.

Input / output format

Input: Raw text utterance

Output: A set of predicted intent labels and a sequence of token-aligned slot labels (e.g., BIO format)

Scoring recipe

def compute_metrics(predictions, gold):
    intent_correct = (set(predictions.intents) == set(gold.intents))
    slot_correct = (predictions.slots == gold.slots)
    overall_acc = 1.0 if (intent_correct and slot_correct) else 0.0
    return overall_acc
# Note: Slot F1 and Intent Acc are computed independently across the dataset using standard token-level and set-matching definitions.

Common pitfalls

• Overall accuracy requires exact match of both intents and slots at the sentence level; partial matches do not count.
• Multiple intent detection requires predicting the exact set of intents; missing or extra intents count as incorrect.
• Slot F1 is computed at the token level, so boundary mismatches (e.g., BIO tagging errors) heavily penalize the score.

Evidence (verbatim from paper)

As for evaluation metrics, following previous works, we adopt accuracy (Acc) for multiple intent detection, F1 score for slot filling, and overall accuracy for the sentence-level semantic frame parsing. Overall accuracy denotes the ratio of sentences whose intents and slots are all correctly predicted.

Citation

@misc{xing2022coguidingnet,
  title={Co-guiding Net: Achieving Mutual Guidances between Multiple Intent Detection and Slot Filling via Heterogeneous Semantics-Label Graphs},
  author={Xing et al. (2022)},
  year={2022},
  note={arXiv:2210.10375}
}

• arXiv: 2210.10375