Bing Chat Open Domain Eval

bing-chat-open-domain-eval

S3-DST: Structured Open-Domain Dialogue Segmentation and State Tracking in the Era of LLMs — Sarkar Snigdha Sarathi Das et al. (2023) (arXiv:2309.08827, 2023)

What this evaluates

Evaluates an LLM's ability to perform zero-shot dialogue segmentation and joint state tracking on real-world, open-domain human-LLM conversations. It probes the model's capacity to identify topic shifts, assign intent and domain labels, and maintain context over long multi-turn interactions without hallucination.

Datasets

• Bing Chat (Internal Human-LLM Dialogue Dataset) — total 484; splits: dev (150), test (334)

Metrics

• JGA (I/D) (primary) — range: [0, 1]
  • Joint Goal Accuracy measuring the proportion of turns where both intent and domain labels are correctly predicted.
• JGA (S/I/D) — range: [0, 1]
  • Joint Goal Accuracy measuring the proportion of turns where segment boundary, intent, and domain labels are all correctly predicted.

Input / output format

Input: Dialogue turns with preceding context, provided in XML-structured format along with system instructions for segmentation and state tracking.

Output: Structured text (XML/SQL-like) specifying segment boundary flags, segment intent, and segment domain for each turn.

Scoring recipe

def compute_jga(predictions, gold):
    correct = 0
    for pred, gold_turn in zip(predictions, gold):
        if pred['intent'] == gold_turn['intent'] and pred['domain'] == gold_turn['domain']:
            correct += 1
    return correct / len(gold)

Common pitfalls

• Model tends to overindex on the current turn's content without considering preceding context, leading to conflicting labels within a coherent segment.
• Longer dialogues cause steep performance degradation in baselines due to context forgetting, requiring explicit grounding mechanisms like PAR.

Evidence (verbatim from paper)

For state tracking, we consider Joint Goal Accuracy (JGA), which measures the proportion of turns for which all state values are correctly inferred. For Bing Chat, we report JGA with just intent and domain (I/D) as these are the true state values of interest, as well as JGA with segment, intent, and domain accuracy (S/I/D) for completeness.

Citation

@misc{das2023s3dst,
  title={S3-DST: Structured Open-Domain Dialogue Segmentation and State Tracking in the Era of LLMs},
  author={Sarkar Snigdha Sarathi Das et al. (2023)},
  year={2023},
  note={arXiv:2309.08827}
}

• arXiv: 2309.08827