Polqa Eval

polqa-eval

PolQA: Polish Question Answering Dataset — Rybak et al. (2022) (arXiv:2212.08897, 2022)

What this evaluates

Evaluates open-domain question answering in Polish by measuring both passage retrieval accuracy and answer generation quality. It probes a model's ability to retrieve relevant evidence from a large corpus and accurately extract or generate answers from those passages.

Datasets

• PolQA — total 7000; splits: train (5000), val (1000), test (1000)

Metrics

• accuracy of top 10 candidates — range: [0, 1]
  • Fraction of questions where the gold evidence passage is present in the top 10 retrieved passages.
• fuzzy_match (primary) — range: [0, 1]
  • For numerical answers: exact match after regex extraction. For non-numerical answers: match if character-wise Levenshtein distance ≤ 50% of the gold answer length. If multiple gold answers exist, the best match is selected.

Input / output format

Input: Retriever: question and candidate passage. Reader: question concatenated with all available relevant passages.

Output: Retriever: ranked list of top 10 passages. Reader: generated token sequence representing the answer.

Scoring recipe

def score_fuzzy_match(pred, golds):
    if is_numerical(golds[0]):
        return pred == extract_num(golds[0])
    best_match = False
    for gold in golds:
        dist = levenshtein_distance(pred, gold)
        if dist <= 0.5 * len(gold):
            best_match = True
            break
    return best_match

Common pitfalls

• Validation and test sets are restricted to passages found via the 'Standard_manual' method to avoid annotation bias, which differs from standard open-domain splits.
• Numerical answers require strict exact match after regex extraction, while non-numerical answers use a lenient 50% Levenshtein threshold, requiring careful handling of answer types.
• The reader is evaluated both on manually labeled passages and on retriever-returned passages, so end-to-end performance can be significantly lower than reader-only performance.

Evidence (verbatim from paper)

During the evaluation, we first encode all Wikipedia passages and index them using FAISS (Johnson et al., 2019). Then, for each question, we retrieve the top 10 most similar passages through an exhaustive search. We measure model performance through the accuracy of top 10 candidates. For numerical answers, we extract the numeral (Arabic or Roman) using regular expression and expect the equality between prediction and true value. For the rest of the questions, we calculate character-wise Levenshtein distance (Levenshtein, 1966), which is allowed to reach 50% of the answer length for a match. In case there is more than one correct answer, we compare the prediction to each and choose the best matching ones.

Citation

@misc{rybak2022polqa,
  title={PolQA: Polish Question Answering Dataset},
  author={Rybak et al. (2022)},
  year={2022},
  note={arXiv:2212.08897}
}

• arXiv: 2212.08897