Dfm Dialogue Eval

dfm-dialogue-eval

DFM: Dialogue Foundation Model for Universal Large-Scale Dialogue-Oriented Task Learning — Zhi Chen et al. (arXiv:2205.12662, 2022)

What this evaluates

Evaluates a unified dialogue foundation model across representation, knowledge distillation, and generation capabilities on diverse dialogue-oriented tasks. It probes the model's ability to perform intent detection, slot filling, semantic parsing, dialogue state tracking, text-to-SQL, and end-to-end task-oriented dialogue generation.

Datasets

• DialoGLUE — total ?; splits: test (-1), dev (-1)
• MULTIWOZ2.0 — total ?; splits: test (-1)
• MULTIWOZ2.2 — total ?; splits: test (-1)
• Spider — total ?; splits: dev (-1)
• CoSQL — total ?; splits: dev (-1)
• CLINC150 — total ?; splits: test (-1)
• BANKING77 — total ?; splits: test (-1)
• HWU64 — total ?; splits: test (-1)
• RESTAURANT8K — total ?; splits: test (-1)
• DSTC8 — total ?; splits: test (-1)
• TOP — total ?; splits: test (-1)
• PERSONALCHAT — total ?; splits: test (-1)
• COQA — total ?; splits: test (-1)
• SAMSUM — total ?; splits: test (-1)
• CANARD — total ?; splits: test (-1)

Metrics

• exact match (EM) (primary) — range: [0, 1]
  • Percentage of predictions that exactly match the gold structured logical form or SQL query.
• accuracy (ACC.) — range: [0, 1]
  • Percentage of correctly predicted intents or labels.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for slot filling predictions.
• joint goal accuracy (JGA) — range: [0, 1]
  • Percentage of dialogue turns where all slot values are predicted correctly.
• BLEU-4 — range: [0, 1]
  • 4-gram precision score measuring n-gram overlap between generated response and reference.
• Combined — range: [0, 1]
  • 0.5*(Inform+Success)+BLEU, where Inform and Success measure task completion constraints and entity matching.

Input / output format

Input: Dialogue context or task instruction formatted as text-to-text prompts.

Output: Structured logical forms (for distillation tasks) or natural language responses (for generation tasks).

Scoring recipe

def score(predictions, gold, task):
    if task in ['intent', 'slot', 'qa', 'text2sql']:
        return exact_match_or_f1(predictions, gold)
    elif task == 'dst':
        return joint_goal_accuracy(predictions, gold)
    elif task in ['generation', 'tod']:
        return bleu4(predictions, gold)
    elif task == 'tod_combined':
        inform = check_constraints(predictions, gold)
        success = check_entities(predictions, gold)
        return 0.5 * (inform + success) + bleu4(predictions, gold)

Common pitfalls

• Mixing up zero-shot (unseen in training) vs fine-tuned settings; DSTC8 and TOP are explicitly noted as unseen/zero-shot.
• Using different decoding strategies (Beam vs PICARD) significantly changes text-to-SQL scores and must be reported consistently.
• DialoGLUE requires preserving specific downstream architectures (e.g., Trippy for DST) during fine-tuning; altering them invalidates comparisons.

Evidence (verbatim from paper)

BLEU is BLEU-4 score, which measures the response consistency. Combined score equals to 0.5*(Inform+Success)+BLEU.

Citation

@misc{chen2022dfm,
  title={DFM: Dialogue Foundation Model for Universal Large-Scale Dialogue-Oriented Task Learning},
  author={Zhi Chen et al.},
  year={2022},
  note={arXiv:2205.12662}
}

• arXiv: 2205.12662