Pat Questions Eval

pat-questions-eval

PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering — Meem et al. (2024) (arXiv:2402.11034, 2024)

What this evaluates

Evaluates large language models' ability to answer present-anchored temporal questions that require up-to-date world knowledge and multi-hop reasoning, such as identifying the current holder of a position or the previous president. It specifically probes performance degradation due to knowledge obsolescence and complex temporal relations.

Datasets

• PAT-Questions — total ?; splits: test (-1)

Metrics

• exact-match accuracy (EM) (primary) — range: [0, 1]
  • 1 if the generated text contains an exact match to the gold answer or vice-versa, else 0. To mitigate phrasing differences, each gold answer is annotated with all possible aliases from Wikidata via SPARQL queries.
• token-level F1 — range: [0, 1]
  • Standard token-level F1 score computed between the generated answer and the gold answer.

Input / output format

Input: Question text. In RAG settings, up to five 300-token Wikipedia document chunks retrieved via Google Custom Search and ranked by BM25 are appended as context.

Output: A concise textual answer to the question, instructed to be 'in a few words'.

Scoring recipe

def compute_metrics(predictions, golds):
    em_scores, f1_scores = [], []
    for pred, gold in zip(predictions, golds):
        aliases = gold.get('aliases', [gold])
        em = 1.0 if any(a in pred or pred in a for a in aliases) else 0.0
        em_scores.append(em)
        p_tokens, g_tokens = set(pred.lower().split()), set(gold.lower().split())
        if not p_tokens or not g_tokens:
            f1 = 0.0
        else:
            prec = len(p_tokens & g_tokens) / len(p_tokens)
            rec = len(p_tokens & g_tokens) / len(g_tokens)
            f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    return sum(em_scores) / len(em_scores), sum(f1_scores) / len(f1_scores)

Common pitfalls

• LLMs may generate factually correct answers with different phrasing, causing strict EM to score them as incorrect unless Wikidata aliases are applied.
• RAG evaluation is strictly compared against December 2023 gold annotations only, as comparing current retrieved documents against 2021 outdated answers is considered illogical.
• Model verbosity can artificially deflate token-level F1 scores even when the core answer is correct.

Evidence (verbatim from paper)

We employ token-level F1 (Rajpurkar et al., 2016) and Chen et al. (2023)’s exact matching (EM) Accuracy metric for the LLMs where if the generated text contains an exact match to the answer or vice-versa, it is considered a correct answer. To address the issue where LLMs might produce an accurate yet differently phrased response to PAT-Questions, such as "Man United" instead of "Manchester United F.C.," resulting in a zero exact match (EM) score, we annotate each answer with all possible aliases from Wikidata using SPARQL queries.

Citation

@misc{meem2024patquestions,
  title={PAT-Questions: A Self-Updating Benchmark for Present-Anchored Temporal Question-Answering},
  author={Meem et al. (2024)},
  year={2024},
  note={arXiv:2402.11034}
}

• arXiv: 2402.11034