Trec Ikat 2023 Eval

trec-ikat-2023-eval

TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview — Aliannejadi et al. (2024) (arXiv:2401.01330, 2024)

What this evaluates

This benchmark evaluates a system's ability to perform personalized conversational search by retrieving relevant passages and generating fluent, grounded responses. It specifically probes how well an agent can adapt its output to user-specific context encoded in a Personal Text Knowledge Base (PTKB) while maintaining provenance traceability.

Datasets

• TREC iKAT 2023 — total 36; splits: train (11), test (25)
• ClueWeb22-B Subset — total 116838987; splits: collection (116838987)

Metrics

• naturalness — range: [0, 4]
  • 0-4 scale assessing if the response sounds human-like and fluent, ranging from 0 (No) to 4 (Yes).
• groundedness (primary) — range: [0, 4]
  • 0-4 scale assessing if the response appropriately references and connects to the provenance passages, ranging from 0 (No) to 4 (Yes).

Input / output format

Input: User utterance, conversation history, Personal Text Knowledge Base (PTKB) statements, and a candidate passage collection.

Output: Ranked list of PTKB statements, ranked list of retrieved passages, and a generated response text.

Scoring recipe

def score(predictions, gold):
    response = predictions['response']
    passages = gold['provenance_passages']
    # GPT-4 evaluates based on track-specified rubrics
    nat_score = call_gpt4(f"Naturalness: Does the response sound human-like?\n{response}")
    grd_score = call_gpt4(f"Groundedness: Does the response reference the passages?\n{response}\nPassages: {passages}")
    return {'naturalness': nat_score, 'groundedness': grd_score}

Common pitfalls

• Evaluating on generic or clarification turns that were explicitly pruned from the assessment pool.
• Assuming traditional IR metrics (e.g., nDCG) are the primary evaluation method, whereas response quality relies on GPT-4 automated scoring.
• Failing to provide provenance passages when requesting groundedness scores, as GPT-4 requires them to assess factual alignment.

Evidence (verbatim from paper)

Each response was then evaluated from two perspectives: groundedness and naturalness. The criteria and the definitions we provided to GPT-4 for the assessment of each were as follows:

Naturalness: Does the response sound human-like?

• 0. No - The response does not sound like something a human would say given the conversation.

• 1. Somewhat - Parts of the response can be generated by human, but it is overall not fluent.

• 2. Slightly natural - The response is almost human-like. The response is well-formed but is not natural.

• 3. Yes (but not completely) - The response is almost human-like. The response is well-formed and natural in most parts but has some parts that are not natural.

• 3. Yes - The response is perfectly human-like and fluent.

Groundedness: Does the response appropriately reference or connect to the information provided in the provenance passages?

Citation

@misc{aliannejadi2024trecikat,
  title={TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview},
  author={Aliannejadi et al. (2024)},
  year={2024},
  note={arXiv:2401.01330}
}

• arXiv: 2401.01330