Dsp Eval

dsp-eval

Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP — Khattab et al. (2022) (arXiv:2212.14024, 2022)

What this evaluates

Evaluates retrieval-augmented language models on open-domain, multi-hop, and conversational question answering by testing their ability to dynamically search for evidence, bootstrap in-context demonstrations, and generate accurate answers without fine-tuning.

Datasets

• Open-SQuAD — total ?; splits: dev (1000), test (-1)
• HotPotQA — total ?; splits: val (1000), train (-1)
• QReCC — total ?; splits: val (400), train (-1)

Metrics

• EM (primary) — range: [0, 1]
  • Exact match accuracy: 1 if the predicted answer string exactly matches the gold answer string, 0 otherwise.
• F1 — range: [0, 1]
  • Token-level F1 score computed as the harmonic mean of precision and recall between the predicted and gold answer tokens.
• nF1 — range: [0, 1]
  • Novel F1 score: token-level F1 overlap between system response and ground truth, discounting common stopwords and terms present in the question or earlier conversation turns.

Input / output format

Input: A question string (for open-domain and multi-hop QA) or a sequence of conversation turns (for conversational QA), provided alongside up to 16 randomly sampled training examples and retrieved Wikipedia passages.

Output: A single predicted answer string (or conversational response), generated via greedy decoding or selected via self-consistency voting over 20 sampled reasoning chains.

Scoring recipe

def compute_metrics(predictions, golds, task='qa'):
    em_scores = [1.0 if p.strip() == g.strip() else 0.0 for p, g in zip(predictions, golds)]
    f1_scores = [token_f1(p, g) for p, g in zip(predictions, golds)]
    if task == 'convqa':
        nf1_scores = [novel_f1(p, g, question) for p, g, question in zip(predictions, golds, questions)]
        return {'EM': sum(em_scores)/len(em_scores), 'F1': sum(f1_scores)/len(f1_scores), 'nF1': sum(nf1_scores)/len(nf1_scores)}
    return {'EM': sum(em_scores)/len(em_scores), 'F1': sum(f1_scores)/len(f1_scores)}

Common pitfalls

• Validation and test sets are subsampled to 1000 questions (or 400 conversations) rather than using the full official splits.
• Results are averaged over 5 random seeds, each using a different 16-shot training subset, introducing sampling variance not always reported.
• nF1 specifically discounts stopwords and question terms, differing from standard conversational F1 metrics.

Evidence (verbatim from paper)

We subsample the validation and test sets to 1000 questions (or 400 conversations, where applicable) and report average quality across five seeds where each seed fixes a single k-shot training set of examples. Table 1 reports the answer EM and F1. We report the novel-F1 metric (nF1; Paranjape et al. 2022), which computes the F1 overlap between the system response and the ground truth while discounting common stopwords and terms present in the question (or earlier questions).

Citation

@misc{khattab2022demonstratesearchpredict,
  title={Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP},
  author={Khattab et al. (2022)},
  year={2022},
  note={arXiv:2212.14024}
}

• arXiv: 2212.14024