Ccl Slu Eval

ccl-slu-eval

Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding — Kim et al. (2024) (arXiv:2405.15097, 2024)

What this evaluates

Evaluates intent classification robustness under noisy ASR conditions by measuring how well a model aligns noisy transcripts with clean references and preserves semantic consistency.

Datasets

• SLURP — total ?; splits: test (-1)
• Timers — total ?; splits: test (-1)
• FSC — total ?; splits: test (-1)
• SNIPS — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted intent labels out of total instances.
• Macro F1-score — range: [0, 1]
  • Unweighted mean of F1 scores computed per intent class, then averaged across all classes.

Input / output format

Input: Noisy or clean ASR transcripts (text strings), optionally tokenized, provided as input to the encoder.

Output: Predicted intent class label.

Scoring recipe

def compute_metrics(preds, golds):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    classes = sorted(set(golds))
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0)
    macro_f1 = sum(f1s) / len(f1s)
    return acc, macro_f1

Common pitfalls

• ASR error rates (WER) vary significantly across datasets; SLURP and Timers have higher WER due to diverse accents, while FSC has the lowest.
• Performance is highly sensitive to noise levels (e.g., Noisy0.24 vs Noisy0.56), and models must explicitly handle insertion, deletion, and substitution errors rather than just learning clean mappings.

Evidence (verbatim from paper)

Table 3: Results for macro F1-score and accuracy performances on SLURP dataset for our model trained with CCL method. We compare the CCL method and other baselines for clean and noisy transcripts.

Citation

@misc{kim2024contrastive,
  title={Contrastive and Consistency Learning for Neural Noisy-Channel Model in Spoken Language Understanding},
  author={Kim et al. (2024)},
  year={2024},
  note={arXiv:2405.15097}
}

• arXiv: 2405.15097