Jaquad Eval

jaquad-eval

JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension — So et al. (2022) (arXiv:2202.01764, 2022)

What this evaluates

Extractive machine reading comprehension in Japanese. It probes a model's ability to locate exact answer spans in Japanese Wikipedia text given a question, evaluating performance across different answer types, question reasoning types, and answer lengths.

Datasets

• JaQuAD — total 39696; splits: dev (-1), test (-1); repo https://github.com/SkelterLabsInc/JaQuAD

Metrics

• F1 score (primary) — range: percent
  • Token-level F1 score measuring the overlap between predicted and gold answer spans. Calculated as 2 * (precision * recall) / (precision + recall), where precision and recall are based on token counts.
• EM — range: percent
  • Exact Match accuracy, where the predicted answer string must exactly equal the gold answer string to receive a score of 1, otherwise 0.

Input / output format

Input: A Japanese question paired with a context paragraph from Japanese Wikipedia.

Output: An exact text span extracted from the context that answers the question.

Scoring recipe

def compute_metrics(preds, golds):
    f1_scores = []
    em_scores = []
    for pred, gold in zip(preds, golds):
        pred_tokens = pred.split()
        gold_tokens = gold.split()
        common = Counter(pred_tokens) & Counter(gold_tokens)
        num_same = sum(common.values())
        if num_same == 0:
            f1_scores.append(0.0)
            em_scores.append(0.0)
            continue
        precision = num_same / len(pred_tokens)
        recall = num_same / len(gold_tokens)
        f1 = (2 * precision * recall) / (precision + recall)
        f1_scores.append(f1)
        em_scores.append(1.0 if pred == gold else 0.0)
    return sum(f1_scores) / len(f1_scores), sum(em_scores) / len(em_scores)

Common pitfalls

• Context truncation to 384 tokens may cut off answer spans, artificially lowering scores.
• Rare answer/question types (e.g., Manner, Cause, Logical reasoning) comprise <1% of data, making aggregate metrics unrepresentative for those categories.
• Short (1-2 tokens) and long (9+ tokens) answers show systematically lower performance, so overall scores mask length-dependent difficulty.

Evidence (verbatim from paper)

The baseline achieves 78.92% for F1 score and 63.38% for EM on test set.

Citation

@misc{so2022jaquad,
  title={JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension},
  author={So et al. (2022)},
  year={2022},
  note={arXiv:2202.01764}
}

• arXiv: 2202.01764