Simmc2.0 Eval

simmc2.0-eval

SimpleMTOD: A Simple Language Model for Multimodal Task-Oriented Dialogue with Symbolic Scene Representation — Hemanthage et al. (2023) (arXiv:2307.04907, 2023)

What this evaluates

Evaluates multimodal task-oriented dialogue capabilities, specifically focusing on dialogue state tracking, disambiguation, coreference resolution, and response generation using visual scene representations.

Datasets

• SIMMC 2.0 — total ?; splits: dev-test (-1), test-standard (-1)

Metrics

• Intent-F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall for predicting the correct dialogue intent class. Computed as 2 * (precision * recall) / (precision + recall).
• Slot-F1 — range: [0, 1]
  • Harmonic mean of precision and recall for predicting dialogue state slot values. Matches predicted slot-value pairs against ground truth.
• Joint Accuracy — range: [0, 1]
  • Exact match accuracy where the entire predicted belief state (all intents and slots) must exactly match the ground truth state.
• Request Slot-F1 — range: [0, 1]
  • F1 score specifically for predicting request slots in the dialogue state.

Input / output format

Input: Multimodal dialogue context $C_t$ including de-localized object tokens, spatial information, and bounding box areas; optionally includes ground-truth system belief state for response generation.

Output: Task-dependent: YES/NO tokens for disambiguation, de-localized object tokens with spatial info for coreference, natural language system responses for generation, and structured belief states (intent, slots) for DST.

Scoring recipe

pred_intent = extract_intent(prediction)
gold_intent = extract_intent(gold)
intent_f1 = f1_score(gold_intent, pred_intent)

pred_slots = extract_slots(prediction)
gold_slots = extract_slots(gold)
slot_f1 = f1_score(gold_slots, pred_slots)

joint_acc = mean([1.0 if p == g else 0.0 for p, g in zip(pred_slots, gold_slots)])

Common pitfalls

• The test-standard split is held-out and not publicly available; final evaluation must use the dev-test split.
• End-to-end vs task-specific setups differ: in E2E mode, disambiguation/coref labels are derived from the predicted system intent, not ground truth.
• Coreference resolution tie-breaking relies on bounding box area when multiple objects share the same de-localized token in a region.

Evidence (verbatim from paper)

We follow the experimental setup of the SIMMC 2.0 challenge with same dataset-splits, inference time limitations, and performance metrics. See Appendix:B for details. It should be noted that the test-standard split of the SIMMC2.0 dataset is not publicly available and is a held-out set for evaluating submissions to SIMMC2.0 challenge. Therefore, the final version of our model could only be evaluated on the dev-test split. Table columns report Intent-F1, Slot-F1, Request Slot-F1, and Joint Accuracy.

Citation

@misc{hemanthage2023simplemtod,
  title={SimpleMTOD: A Simple Language Model for Multimodal Task-Oriented Dialogue with Symbolic Scene Representation},
  author={Hemanthage et al. (2023)},
  year={2023},
  note={arXiv:2307.04907}
}

• arXiv: 2307.04907