Ipqa Eval

ipqa-eval

IPQA: A Benchmark for Core Intent Identification in Personalized Question Answering — Jieyong Kim et al. (2025) (arXiv:2510.23536, 2025)

What this evaluates

This benchmark evaluates a model's ability to identify core user intents in personalized question answering. It probes whether systems can infer prioritized motivations from a user's historical Q&A interactions and a target question's narrative, rather than relying on explicit user statements.

Datasets

• IPQA — total 7730; splits: train (4655), val (1530), test (1545)

Metrics

• IPQA-Eval F1 (primary) — range: [0, 1]
  • Precision, recall, and F1 are computed based on LLM-based alignment between predicted core intents and ground truth intents. Each predicted intent is matched to the best-aligned ground truth intent. A ground truth intent is counted as correctly identified only once. Precision = |correctly identified GT intents| / |predicted intents|, Recall = |correctly identified GT intents| / |GT intents|, F1 = 2 * (Precision * Recall) / (Precision + Recall).

Input / output format

Input: A target question q and a chronological user profile P_u consisting of historical pairs of questions and source information (narrative + selected answer).

Output: A set of predicted core intents, where each intent comprises an intent name, a description, and a reference text excerpt.

Scoring recipe

def ipqa_eval(predicted_intents, ground_truth_intents):
    matched_gt = set()
    for pred in predicted_intents:
        best_match = llm_find_best_alignment(pred, ground_truth_intents)
        if best_match is not None:
            matched_gt.add(best_match)
    tp = len(matched_gt)
    precision = tp / len(predicted_intents) if predicted_intents else 0
    recall = tp / len(ground_truth_intents) if ground_truth_intents else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return precision, recall, f1

Common pitfalls

• One-to-many comparison reduces computational cost but may cause multiple predicted intents to map to the same ground truth intent, which only counts once for recall.
• LLM-based matching introduces potential bias or inconsistency compared to exact string matching, requiring careful prompt design for the evaluator.
• Intents are multi-component (name, description, reference), so evaluators must align on semantic meaning rather than exact text.

Evidence (verbatim from paper)

Based on these alignments, we compute precision, recall, and F1 to measure intent identification performance.

Citation

@misc{kim2025ipqa,
  title={IPQA: A Benchmark for Core Intent Identification in Personalized Question Answering},
  author={Jieyong Kim et al. (2025)},
  year={2025},
  note={arXiv:2510.23536}
}

• arXiv: 2510.23536