Toqb Eval

toqb-eval

The Task-oriented Queries Benchmark (ToQB) — Yim et al. (2024) (arXiv:2406.02943, 2024)

What this evaluates

Evaluates a model's ability to perform dialogue user request summarization, specifically extracting and condensing a user's intent and key constraints from a multi-turn task-oriented conversation into a single paragraph.

Datasets

• ToQB (Task-oriented Queries Benchmark) — total 2922; splits: test (-1); repo https://github.com/google/task-oriented-queries

Metrics

• key_slot_verification (primary) — range: other
  • Automatic verification checks whether key slots from the original dialogue annotations are correctly captured in the generated summary. Manual reviewers cross-check annotations and identify hallucinations for unsuccessful responses. Lexical, syntactic, and semantic analyses are also conducted on the final dataset.

Input / output format

Input: A sequence of transcribed user and system utterances representing a task-oriented dialogue, optionally accompanied by domain and key slot annotations.

Output: A one-paragraph summary capturing the user's request, intent, and key constraints (e.g., locations, times, conditional preferences).

Scoring recipe

def evaluate(dialogue, summary, annotations):
    # 1. Extract key slots from dialogue annotations
    slots = extract_slots(annotations)
    # 2. Verify key slots are correctly summarized
    slot_match = verify_slots_in_text(summary, slots)
    # 3. Check for hallucinations using dedicated tools
    hallucination = detect_hallucinations(summary)
    # 4. Manual review for unsuccessful responses
    if hallucination or not slot_match:
        return 'unsuccessful'
    return 'successful'

Common pitfalls

• Summaries may incorrectly include system responses or situational context instead of focusing solely on the user's intent.
• Conditional user requests (e.g., fallback preferences) are often oversimplified or lost in the summary.
• Automatic slot verification fails if the original dialogue lacks explicit annotations.

Evidence (verbatim from paper)

The automatic verifications are conducted if and only if the annotations extracted from the input dialogue data are available. Using the annotations, automatic verifications, for example, check whether the key slots are correctly summarized in given responses [31]. The manual reviewers can also utilize the annotations to expedite the cross-checking and classification processes. The manual reviewers are responsible for identifying hallucinations for unsuccessful responses.

Citation

@misc{yim2024toqb,
  title={The Task-oriented Queries Benchmark (ToQB)},
  author={Yim et al. (2024)},
  year={2024},
  note={arXiv:2406.02943}
}

• arXiv: 2406.02943