Drcd Eval

drcd-eval

DRCD: a Chinese Machine Reading Comprehension Dataset — Shao et al. (2018) (arXiv:1806.00920, 2018)

What this evaluates

Evaluates a model's ability to perform span-based machine reading comprehension in traditional Chinese. It probes factual retrieval and exact answer extraction from provided context paragraphs without requiring complex inference or multiple-choice reasoning.

Datasets

• DRCD — total 33941; splits: dev (-1), test (-1); repo https://github.com/DRCKnowledgeTeam/DRCD

Metrics

• F1 score (primary) — range: [0, 1]
  • Character-level F1 score calculated as the intersection of predicted and ground-truth Chinese characters divided by the average of their lengths. Punctuation is ignored.
• Exact Match — range: [0, 1]
  • Binary metric that returns 1 if the predicted answer string exactly matches the ground-truth answer string (ignoring punctuation), else 0.

Input / output format

Input: A context paragraph and a corresponding question in traditional Chinese.

Output: A text span extracted from the context paragraph that answers the question.

Scoring recipe

def compute_metrics(pred, gold):
    pred_clean = remove_punctuation(pred)
    gold_clean = remove_punctuation(gold)
    em = 1.0 if pred_clean == gold_clean else 0.0
    pred_chars = set(pred_clean)
    gold_chars = set(gold_clean)
    if not pred_chars or not gold_chars:
        f1 = 0.0
    else:
        precision = len(pred_chars & gold_chars) / len(pred_chars)
        recall = len(pred_chars & gold_chars) / len(gold_chars)
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1, em

Common pitfalls

• Punctuation must be stripped before comparison, otherwise scores will be artificially low.
• F1 is computed at the character level (bag of characters), not word or token level, which is critical for Chinese text.
• Models must extract a contiguous span from the context; generated free-form answers are not evaluated correctly.

Evidence (verbatim from paper)

F1 score and exact match from Rajpurkar et al. (2016) are used as the evaluation metrics. Both metrics ignore punctuations. In F1 score metric, we consider predictions and ground truth as bag of Chinese character.

Citation

@misc{shao2018drcd,
  title={DRCD: a Chinese Machine Reading Comprehension Dataset},
  author={Shao et al. (2018)},
  year={2018},
  note={arXiv:1806.00920}
}

• arXiv: 1806.00920