# Squad Fquad Eval

> Evaluates a model's ability to extract and rank answer candidates from a given context for phrase-indexed question answering. It probes both the quality of candidate retrieval and the accuracy of final answer selection against gold spans. Use when the user wants to benchmark on SQuAD v1.1, FQuAD, or asks about evaluating this task. Reports exact-match.

- Skill: `qhjqhj00/squad-fquad-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/squad-fquad-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/squad-fquad-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/squad-fquad-eval

---


# squad-fquad-eval

> EfficientQA : a RoBERTa Based Phrase-Indexed Question-Answering System — Chaybouti et al. (2021) (arXiv:2101.02157, 2021)

## What this evaluates

Evaluates a model's ability to extract and rank answer candidates from a given context for phrase-indexed question answering. It probes both the quality of candidate retrieval and the accuracy of final answer selection against gold spans.

## Datasets

- **SQuAD v1.1** — total 100000; splits: train (87599), dev (10570); HF `squad`
- **FQuAD** — total 26399; splits: train (20731), dev (5668); HF `fquad`

## Metrics

- `exact-match` **(primary)** — range: [0, 1]
  - 1 if the predicted answer string exactly matches any of the gold answer strings, else 0.
- `f1-score` — range: [0, 1]
  - Token-level F1 score between the predicted answer and the best-matching gold answer string.

## Input / output format

**Input**: A natural language question and a context passage.

**Output**: A ranked list of extracted answer candidates (phrases) from the context; the top-ranked candidate is taken as the final prediction.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    em, f1 = 0.0, 0.0
    for pred, gold_list in zip(predictions, golds):
        em += 1.0 if pred in gold_list else 0.0
        pred_tok = pred.split()
        f1 += max(token_f1(pred_tok, g.split()) for g in gold_list)
    return em / len(predictions), f1 / len(predictions)
```

## Common pitfalls

- The paper evaluates exact-match and F1 over *all* extracted candidates rather than just the top-1 prediction, which measures retrieval recall rather than final answer accuracy.
- Direct comparison with vanilla fine-tuned QA baselines ignores the fundamental difference between span-extraction and phrase-indexed retrieval paradigms.

## Evidence (verbatim from paper)

> Our model was trained on the train set ($87599$ pairs) and evaluated on the development set ($10570$ pairs). The architectures are evaluated with exact-match and f1-score over all selected candidates.

## Citation

```bibtex
@misc{chaybouti2021efficientqa,
  title={EfficientQA : a RoBERTa Based Phrase-Indexed Question-Answering System},
  author={Chaybouti et al. (2021)},
  year={2021},
  note={arXiv:2101.02157}
}
```

- arXiv: 2101.02157