Repliqa Eval

repliqa-eval

RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content — Monteiro et al. (2024) (arXiv:2406.11811, 2024)

What this evaluates

Evaluates LLMs' ability to read unseen reference documents and answer questions based solely on the provided context, as well as their ability to detect unanswerable questions and classify document topics. It specifically probes whether models rely on pre-training memory versus actual context-conditional reading and retrieval skills.

Datasets

• RepLiQA — total 90000; splits: RepLiQA0 (-1); repo https://github.com/ServiceNow/repliqa
• TriviaQA — total ?; splits: test (-1); HF mandarjoshi/trivia_qa

Metrics

• recall (primary) — range: [0, 1]
  • True positives divided by the sum of true positives and false negatives (TP / (TP + FN)). Measures the fraction of correct answers successfully retrieved from the provided context.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall over the 17 candidate document topics. Standard micro-averaged F1 for multi-class classification.
• unanswerable detection rate — range: [0, 1]
  • Percentage of questions where the ground truth is 'unanswerable' and the model correctly outputs 'unanswerable'.

Input / output format

Input: A user query/question paired with a reference context document. For unanswerable questions, the document explicitly lacks the answer. For topic retrieval, a context document plus a list of 17 candidate topics passed through the prompt.

Output: For QA: the answer string or the exact token 'unanswerable'. For topic retrieval: one of the 17 candidate topic names.

Scoring recipe

# QA Recall
tp = sum(1 for p, g in zip(preds, golds) if p == g)
fn = sum(1 for p, g in zip(preds, golds) if p != g and g != "unanswerable")
recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0

# Topic Retrieval F1
# Standard micro-averaged F1 over 17 candidate topics
f1 = compute_micro_f1(preds, golds)

# Unanswerable Detection Rate
correct_refusals = sum(1 for p, g in zip(preds, golds) if g == "unanswerable" and p == "unanswerable")
total_unanswerable = sum(1 for g in golds if g == "unanswerable")
detection_rate = correct_refusals / total_unanswerable if total_unanswerable > 0 else 0.0

Common pitfalls

• Models may rely heavily on pre-training memorization rather than reading the provided context, especially on factual datasets like TriviaQA, leading to inflated performance that masks poor reading capabilities.
• Unanswerable questions require explicit refusal instructions; models frequently hallucinate answers even when prompted to refuse, making detection rates highly prompt-sensitive.
• Removing context can sometimes improve performance on memorizable datasets due to reduced distraction, which skews the evaluation of true context-conditional reading ability.

Evidence (verbatim from paper)

For question answering, we follow the evaluation protocol by Adlakha et al. ([2023]) and measure performance metrics such as the F1 score and recall. Around 20% of questions in RepLiQA are not answerable from provided documents, in which situation we expect (and prompt) models to reply with unanswerable. We further evaluate models in terms of their ability to detect such situations and refuse to reply.

Citation

@misc{monteiro2024repliqa,
  title={RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content},
  author={Monteiro et al. (2024)},
  year={2024},
  note={arXiv:2406.11811}
}

• arXiv: 2406.11811