Silicone Eval

silicone-eval

Hierarchical Pre-training for Sequence Labelling in Spoken Dialog — Chapuis et al. (2020) (arXiv:2009.11152, 2020)

What this evaluates

Evaluates a model's ability to perform sequence labelling on spoken dialogues, specifically predicting dialog acts (DA) and emotion/sentiment (E/S) labels per utterance within multi-utterance conversations.

Datasets

• SILICONE — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly predicted labels out of the total number of utterance-level predictions. Reported as average accuracy across tasks or subgroups (Avg DA, Avg E/S).

Input / output format

Input: Multi-utterance spoken dialog conversations (sequences of utterances).

Output: Per-utterance sequence of predicted labels (Dialog Act or Emotion/Sentiment category).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

Common pitfalls

• Using sequential decoders (CRF/GRU) instead of the recommended plain MLP decoder, which can lead to unfair comparisons or poor performance due to limited training data.
• Ignoring the multi-utterance hierarchical structure of the input, which is critical for capturing discourse-level dependencies in spoken dialog.
• Assuming equal difficulty across tasks; E/S tasks consistently underperform DA tasks due to differences in corpus size and utterances-per-label ratios.

Evidence (verbatim from paper)

Table 4 provides an exhaustive comparison of the different encoders over the SILICONE benchmark. As previously discussed, we adopt a plain MLP as a decoder to compare the different encoders. We show that SILICONE covers a set of challenging tasks as the best performing model achieves an average accuracy of 74.3.

Citation

@misc{chapuis2020hierarchical,
  title={Hierarchical Pre-training for Sequence Labelling in Spoken Dialog},
  author={Chapuis et al. (2020)},
  year={2020},
  note={arXiv:2009.11152}
}

• arXiv: 2009.11152