Lamp QA Eval

lamp-qa-eval

LaMP-QA: A Benchmark for Personalized Long-form Question Answering — Salemi et al. (2025) (arXiv:2506.00137, 2025)

What this evaluates

This benchmark evaluates a model's ability to generate long-form, personalized question-answering responses by aligning outputs with fine-grained, user-specific information needs extracted from community Q&A narratives. It probes aspect-based response quality rather than binary correctness, measuring how well generated answers address individual criteria tailored to a specific user's profile.

Datasets

• LaMP-QA — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/LaMP-Benchmark/LaMP-QA

Metrics

• aspect-based evaluation (primary) — range: [0, 1]
  • Scores responses based on how well they address individual aspects extracted from the user's stated information needs (narratives). The LLM evaluator assigns a continuous score between 0 and 1 for each aspect, and the final metric is the average across all extracted aspects.

Input / output format

Input: Question, user narrative/profile context, retrieved profile items (k=10), and the generated response.

Output: A continuous score between 0 and 1 representing the degree to which the response addresses the user's aspect-based information needs.

Scoring recipe

def compute_aspect_based_score(question, narrative, response):
    aspects = extract_aspects_from_narrative(narrative) # LLM extracts fine-grained criteria
    scores = []
    for aspect in aspects:
        score = llm_rater_score(response, aspect) # LLM assigns 0-1 alignment score
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• Pairwise LLM evaluation suffers from severe position bias, with preference reversal rates up to 78% when response order is swapped.
• Smaller evaluator LLMs (e.g., 0.5B) act as binary classifiers, inflating scores and misaligning with human judgments compared to larger models (32B).
• Using randomly sampled user profiles instead of the actual asker's profile degrades performance below non-personalized baselines, highlighting the necessity of true user-specific context.

Evidence (verbatim from paper)

Lastly, we use our proposed evaluation method in Section[3.2], which scores responses based on how well they address the individual aspects extracted from the user’s information needs. The implementation details for all evaluation approaches are provided in Appendix[C].

Citation

@misc{salemi2025lampqa,
  title={LaMP-QA: A Benchmark for Personalized Long-form Question Answering},
  author={Salemi et al. (2025)},
  year={2025},
  note={arXiv:2506.00137}
}

• arXiv: 2506.00137