Snips Slu Eval

snips-slu-eval

Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces — Coucke et al. (2018) (arXiv:1805.10190, 2018)

What this evaluates

Evaluates end-to-end spoken language understanding by measuring how well an embedded system extracts intents and slots from spoken audio. It probes the pipeline's ability to generalize to unseen queries and handle real-world ASR errors under strict resource constraints.

Datasets

• SmartLights — total ?; splits: train (-1), test (-1)
• Weather — total ?; splits: train (-1), test (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall. Computed separately for intent classification and slot filling, then averaged across intents/slots.
• Precision — range: [0, 1]
  • Ratio of correctly predicted intents/slots to all predicted intents/slots.
• Recall — range: [0, 1]
  • Ratio of correctly predicted intents/slots to all ground truth intents/slots.

Input / output format

Input: Raw audio utterances corresponding to user queries for the SmartLights or Weather domains.

Output: Predicted intent label and associated slot-value pairs.

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions and gold are lists of dicts with 'intent' and 'slots' keys
    # slots are dicts mapping slot_name to value
    correct_intent = sum(1 for p, g in zip(predictions, gold) if p['intent'] == g['intent'])
    intent_prec = correct_intent / len(predictions)
    intent_rec = correct_intent / len(gold)
    intent_f1 = 2 * intent_prec * intent_rec / (intent_prec + intent_rec)
    
    slot_tp, slot_fp, slot_fn = 0, 0, 0
    for p, g in zip(predictions, gold):
        for slot_name in set(p['slots'].keys()) | set(g['slots'].keys()):
            p_val = p['slots'].get(slot_name)
            g_val = g['slots'].get(slot_name)
            if p_val == g_val: slot_tp += 1
            elif p_val is not None and g_val is None: slot_fp += 1
            elif p_val is None and g_val is not None: slot_fn += 1
    slot_prec = slot_tp / (slot_tp + slot_fp) if (slot_tp + slot_fp) > 0 else 0
    slot_rec = slot_tp / (slot_tp + slot_fn) if (slot_tp + slot_fn) > 0 else 0
    slot_f1 = 2 * slot_prec * slot_rec / (slot_prec + slot_rec)
    return {'intent_f1': intent_f1, 'slot_f1': slot_f1, 'intent_prec': intent_prec, 'intent_rec': intent_rec, 'slot_prec': slot_prec, 'slot_rec': slot_rec}

Common pitfalls

• ASR errors propagate to the NLU stage, lowering end-to-end F1 compared to NLU-only F1 on ground-truth text.
• The evaluation uses crowdsourced spoken test data, which may differ in distribution and noise from synthetic training utterances.
• Confidence scoring thresholds are tuned to favor precision over recall, so high recall may indicate poor rejection thresholds.

Evidence (verbatim from paper)

We are interested in computing end-to-end metrics quantifying the ability of the assistants to extract intent and slots from spoken utterances. We create a test set by crowdsourcing a spoken corpus corresponding to the queries of each dataset. For each sentence of the speech corpus, we apply the ASR engine followed by the NLU engine, and compare the predicted output to the ground true intent and slots in the dataset. In the following, we present our results in terms of the classical precision, recall, and F1 scores.

Citation

@misc{coucke2018snips,
  title={Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces},
  author={Coucke et al. (2018)},
  year={2018},
  note={arXiv:1805.10190}
}

• arXiv: 1805.10190