Wikichat Simulated Dialogue Eval

wikichat-simulated-dialogue-eval

WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia — Semnani et al. (2023) (arXiv:2305.14292, 2023)

What this evaluates

Evaluates the factual accuracy, conversational quality, and latency of knowledge-grounded chatbots in simulated multi-turn dialogues across head, tail, and recent knowledge domains.

Datasets

• Simulated Dialogues (WikiChat) — total ?; splits: head (-1), tail (-1), recent (-1); repo https://github.com/stanford-oval/WikiChat

Metrics

• factual_accuracy (primary) — range: percent
  • Percentage of generated claims supported by the Wikipedia knowledge corpus. Calculated as (number of claims verified as true by majority of 3 crowdworkers) / (total claims generated).

Input / output format

Input: Simulated user prompt containing only the title and first sentence of a Wikipedia article, plus the conversation history up to the current turn.

Output: Chatbot response text per turn.

Scoring recipe

total_claims = 0
supported_claims = 0
for response in bot_responses:
    claims = extract_claims(response)
    for claim in claims:
        total_claims += 1
        judgments = get_crowdworker_judgments(claim, wikipedia_corpus) # 3 workers
        if majority_agree(claim, judgments):
            supported_claims += 1
return supported_claims / total_claims

Common pitfalls

• Evaluating only on head knowledge masks severe hallucination in tail and recent knowledge domains.
• Factuality is measured per-claim rather than per-response, requiring accurate claim extraction before evaluation.
• Simulated users driven by GPT-4 may not perfectly replicate human knowledge gaps or conversational behavior.

Evidence (verbatim from paper)

We define the factual accuracy of a chatbot to be the percentage of claims the bot makes in a given dialogue set, that are supported by the knowledge corpus. As mentioned in Section 5.2, this is done by obtaining per-claim judgments of factuality from crowdworkers. We obtain 3 judgements for each of the 5974 claims our chatbots output in total.

Citation

@misc{semnani2023wikichat,
  title={WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia},
  author={Semnani et al. (2023)},
  year={2023},
  note={arXiv:2305.14292}
}

• arXiv: 2305.14292