# Jaquad Eval

> Extractive machine reading comprehension in Japanese. It probes a model's ability to locate exact answer spans in Japanese Wikipedia text given a question, evaluating performance across different answer types, question reasoning types, and answer lengths. Use when the user wants to benchmark on JaQuAD, or asks about evaluating this task. Reports F1 score.

- Skill: `qhjqhj00/jaquad-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/jaquad-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/jaquad-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/jaquad-eval

---


# jaquad-eval

> JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension — So et al. (2022) (arXiv:2202.01764, 2022)

## What this evaluates

Extractive machine reading comprehension in Japanese. It probes a model's ability to locate exact answer spans in Japanese Wikipedia text given a question, evaluating performance across different answer types, question reasoning types, and answer lengths.

## Datasets

- **JaQuAD** — total 39696; splits: dev (-1), test (-1); repo https://github.com/SkelterLabsInc/JaQuAD

## Metrics

- `F1 score` **(primary)** — range: percent
  - Token-level F1 score measuring the overlap between predicted and gold answer spans. Calculated as 2 * (precision * recall) / (precision + recall), where precision and recall are based on token counts.
- `EM` — range: percent
  - Exact Match accuracy, where the predicted answer string must exactly equal the gold answer string to receive a score of 1, otherwise 0.

## Input / output format

**Input**: A Japanese question paired with a context paragraph from Japanese Wikipedia.

**Output**: An exact text span extracted from the context that answers the question.

## Scoring recipe

```python
def compute_metrics(preds, golds):
    f1_scores = []
    em_scores = []
    for pred, gold in zip(preds, golds):
        pred_tokens = pred.split()
        gold_tokens = gold.split()
        common = Counter(pred_tokens) & Counter(gold_tokens)
        num_same = sum(common.values())
        if num_same == 0:
            f1_scores.append(0.0)
            em_scores.append(0.0)
            continue
        precision = num_same / len(pred_tokens)
        recall = num_same / len(gold_tokens)
        f1 = (2 * precision * recall) / (precision + recall)
        f1_scores.append(f1)
        em_scores.append(1.0 if pred == gold else 0.0)
    return sum(f1_scores) / len(f1_scores), sum(em_scores) / len(em_scores)
```

## Common pitfalls

- Context truncation to 384 tokens may cut off answer spans, artificially lowering scores.
- Rare answer/question types (e.g., Manner, Cause, Logical reasoning) comprise <1% of data, making aggregate metrics unrepresentative for those categories.
- Short (1-2 tokens) and long (9+ tokens) answers show systematically lower performance, so overall scores mask length-dependent difficulty.

## Evidence (verbatim from paper)

> The baseline achieves 78.92% for F1 score and 63.38% for EM on test set.

## Citation

```bibtex
@misc{so2022jaquad,
  title={JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension},
  author={So et al. (2022)},
  year={2022},
  note={arXiv:2202.01764}
}
```

- arXiv: 2202.01764