Dstc11 Track2 Intent Induction Eval

dstc11-track2-intent-induction-eval

Multi-Stage Coarse-to-Fine Contrastive Learning for Conversation Intent Induction — Chu et al. (2023) (arXiv:2303.05034, 2023)

What this evaluates

Evaluates a model's ability to automatically induce conversation intents by clustering utterances from task-oriented dialogues without prior intent labels. It measures how well the induced clusters align with ground-truth intent categories using supervised clustering metrics.

Datasets

• DSTC11 Track 2 — total ?; splits: development (-1), test-banking (-1), test-finance (-1)

Metrics

• ACC (primary) — range: percent
  • Accuracy of cluster-to-intent alignment after optimal 1:1 matching via the Hungarian algorithm. Used as the primary metric for ranking system submissions.
• NMI — range: [0, 1]
  • Normalized Mutual Information between predicted clusters and reference intents after alignment.
• F1-score — range: [0, 1]
  • F1-score computed on the aligned cluster-to-intent mapping.
• Recall — range: [0, 1]
  • Recall of the aligned cluster-to-intent mapping.
• Precision — range: [0, 1]
  • Precision of the aligned cluster-to-intent mapping.
• ARI — range: [-1, 1]
  • Adjusted Rand Index between predicted clusters and reference intents.

Input / output format

Input: Sequences of utterances from human-to-human customer-agent dialogues, specifically filtered for utterances with non-empty intent and 'InformIntent' dialogue acts.

Output: A cluster label assigned to each input utterance.

Scoring recipe

def score(predictions, gold, task):
    # predictions: list of cluster IDs per utterance
    # gold: list of reference intent labels per utterance
    if task == 1:
        gold_labels = gold  # turn-level reference intents
    else:
        gold_labels = get_classifier_predictions_on_held_out(gold) # Task 2 specific
    mapping = hungarian_algorithm(predictions, gold_labels)
    aligned_preds = [mapping[p] for p in predictions]
    acc = sum(p == g for p, g in zip(aligned_preds, gold_labels)) / len(gold_labels)
    nmi = normalized_mutual_info(gold_labels, predictions)
    prec, rec, f1 = precision_recall_f1(gold_labels, aligned_preds)
    ari = adjusted_rand_score(gold_labels, predictions)
    return {'ACC': acc, 'NMI': nmi, 'F1': f1, 'Recall': rec, 'Precision': prec, 'ARI': ari}

Common pitfalls

• Metrics require an automatic alignment step (Hungarian algorithm) between predicted clusters and reference intents; reporting raw cluster-to-label accuracy without alignment is invalid.
• Task 1 uses turn-level reference labels for alignment, while Task 2 uses classifier predictions on held-out utterances; mixing these protocols will yield incorrect scores.
• The primary ranking metric is ACC, not F1 or NMI, despite all six being reported.

Evidence (verbatim from paper)

Task 1 and task 2 are both evaluated by the following six metrics.: accuracy (ACC) (Huang et al., 2014), normalized mutual information (NMI), F1-score, Recall, Precision, and adjusted rand index (ARI). But the ACC is the primary metric used for ranking system submissions. Metrics dependent on reference intents will be computed using an automatic alignment of cluster labels to reference intent labels. For task 1, alignments will be computed based on turn-level reference intent labels. For task 2, to avoid the need to assign labels to turns in the input transcripts, alignments will be computed using classifier predictions on the set of utterances held out for evaluation. In both cases, 1:1 alignments between induced intents and reference intents will be computed using the Hungarian algorithm (Kuhn, 1955).

Citation

@misc{chu2023intentinduction,
  title={Multi-Stage Coarse-to-Fine Contrastive Learning for Conversation Intent Induction},
  author={Chu et al. (2023)},
  year={2023},
  note={arXiv:2303.05034}
}

• arXiv: 2303.05034