Germanquad Eval

germanquad-eval

GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval — Möller et al. (2021) (arXiv:2104.12741, 2021)

What this evaluates

Evaluates extractive question answering and dense passage retrieval capabilities in German. It probes a model's ability to locate precise answer spans within a given context and retrieve relevant passages from a large corpus.

Datasets

• GermanQuAD — total ?; splits: train (-1), test (-1)

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • Binary score: 1 if the predicted answer span exactly matches the ground truth span, else 0.
• F1 — range: [0, 1]
  • F1 score calculated as the ratio of overlapping words between the predicted answer span and the ground truth answer span.
• Top-1-Accuracy — range: [0, 1]
  • Binary hit-or-miss score: 1 if there is any positional overlap between the model's predicted answer span and the ground truth, else 0.
• Recall@10 — range: percent
  • Percentage of queries for which the correct passage appears in the top 10 retrieved results.

Input / output format

Input: Question and a context passage (for QA); Query and a set of candidate passages (for retrieval).

Output: Extractive answer span (start and end indices or text) for QA; Ranked list of passages for retrieval.

Scoring recipe

def compute_metrics(predictions, golds):
    em = [1.0 if p == g else 0.0 for p, g in zip(predictions, golds)]
    f1 = [f1_overlap(p, g) for p, g in zip(predictions, golds)]
    top1 = [1.0 if has_positional_overlap(p, g) else 0.0 for p, g in zip(predictions, golds)]
    return {
        'EM': sum(em) / len(em),
        'F1': sum(f1) / len(f1),
        'Top-1-Acc': sum(top1) / len(top1)
    }

Common pitfalls

• Exact Match and F1 penalize semantically valid but stylistically different answers (e.g., 'Lion' vs 'in star constellation Lion').
• Top-N-Accuracy is lenient and may mask failures in precise span extraction, making it unsuitable for datasets requiring strict annotation alignment.
• Machine-translated training data yields lower performance than hand-annotated in-language data, despite similar lexical overlap.

Evidence (verbatim from paper)

Next to exact match (EM) and the ratio of overlapping words (F1) metric we also report Top-N-Accuracy scores. Top-N-Accuracy is a binary hit or miss score for the first N model predictions. The value is one if there is any positional overlap between the ground-truth answer and the model prediction — otherwise, it is zero. Such a hit or miss score is less prone to details of how answers are annotated (e.g., Answer1: "19th century", Answer2: "middle of 19th century") and useful in settings without multi-way annotations or where answers can be more than numbers, single entities, or a few words.

Citation

@misc{moeller2021germanquad,
  title={GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval},
  author={Möller et al. (2021)},
  year={2021},
  note={arXiv:2104.12741}
}

• arXiv: 2104.12741