Query Disambiguation Afc Eval

query-disambiguation-afc-eval

Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity's Last Exam — Majurski et al. (2026) (arXiv:2603.04454, 2026)

What this evaluates

Evaluates whether rewriting ambiguous queries using answer-free context improves factual QA accuracy compared to standard RAG baselines. It probes a model's ability to leverage disambiguated queries for better retrieval-augmented generation and measures the semantic alignment between rewritten queries and grounding contexts.

Datasets

• HLE-subset — total ?; splits: test (-1)
• flashrag_fermi — total ?; splits: test (-1)
• ai_plan — total ?; splits: test (-1)
• arXiv_2502_17521v1 — total ?; splits: test (-1)

Metrics

• benchmark accuracy (primary) — range: percent
  • Percentage of correctly answered questions. Calculated as (number of correct answers / total questions) * 100.
• cosine similarity — range: [-1, 1]
  • Cosine similarity between the embedding of the query (original or rewritten) and the grounding context, measured using e5-mistral-7b-instruct.

Input / output format

Input: Per instance: (1) Original question, (2) Original question with Answer-Free Context prepended, (3) Rewritten question (AFC withheld), or (4) Rewritten question with AFC. For in-situ baseline, a prompt instructing the model to rewrite the question using AFC before answering.

Output: A textual answer to the benchmark question. For extractive QA, the exact answer string; for generative benchmarks, a generated response.

Scoring recipe

def compute_accuracy(predictions, gold_answers):
    correct = sum(1 for p, g in zip(predictions, gold_answers) if normalize(p) == normalize(g))
    return (correct / len(gold_answers)) * 100

def compute_cosine_similarity(query, context, model):
    q_emb = model.encode(query)
    c_emb = model.encode(context)
    return np.dot(q_emb, c_emb) / (np.linalg.norm(q_emb) * np.linalg.norm(c_emb))

Common pitfalls

• Assuming that prepending answer-free context (Orig_Q+AFC) is equivalent to rewriting the query; the paper shows rewriting yields significantly higher accuracy than simple context prepending.
• Confusing the 'in-situ' CoT rewriting baseline with the two-phase rewrite-then-answer paradigm; the paper finds that combining rewrite and answer in a single pass negates the accuracy gains.
• Overgeneralizing results to reasoning-heavy benchmarks; the paper notes that rewriting primarily helps factual recall, while complex reasoning tasks benefit more from raw context inclusion.

Evidence (verbatim from paper)

Benchmark accuracy improved by an average of 0.1346. ... We observe a strong positive relationship: Rewritten questions systematically exhibit higher cosine similarity to the context than the original questions.

Citation

@misc{majurski2026querydisambiguation,
  title={Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity's Last Exam},
  author={Majurski et al. (2026)},
  year={2026},
  note={arXiv:2603.04454}
}

• arXiv: 2603.04454