Qrcd Eval

qrcd-eval

DTW at Qur'an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domain — Premasiri et al. (2022) (arXiv:2205.06025, 2022)

What this evaluates

Evaluates machine reading comprehension on a low-resource religious domain (Qur'an). It probes a model's ability to extract precise answer spans from Arabic text given a question, testing both exact matching and partial semantic/token overlap.

Datasets

• QRCD — total ?; splits: dev (-1), test (-1); repo https://github.com/DamithDR/QuestionAnswering

Metrics

• pRR (primary) — range: [0, 1]
  • Partial Reciprocal Rank. For each query, rank predictions by confidence. Score is 1/rank if the prediction partially matches any gold answer, else 0. Averaged over all queries.
• EM — range: [0, 1]
  • Exact Match. Binary score: 1 if the top predicted answer exactly matches at least one gold answer, 0 otherwise.
• F1@1 — range: [0, 1]
  • Token-level F1 score computed between the top predicted answer and the best-matching gold answer.

Input / output format

Input: Arabic context (Qur'anic verse or paragraph) paired with a question.

Output: A single text span extracted from the context as the predicted answer.

Scoring recipe

def compute_metrics(predictions, golds):
    prr, em, f1 = 0.0, 0.0, 0.0
    for pred, gold_list in zip(predictions, golds):
        # pRR: 1/rank of first partial match
        for i, p in enumerate(pred, 1):
            if partial_match(p, gold_list):
                prr += 1.0 / i
                break
        # EM: exact match on top prediction
        em += 1.0 if exact_match(pred[0], gold_list) else 0.0
        # F1@1: token F1 on top prediction vs best gold
        f1 += token_f1(pred[0], best_matching_gold(pred[0], gold_list))
    n = len(predictions)
    return prr/n, em/n, f1/n

Common pitfalls

• Metrics are computed exclusively on the top-ranked prediction, not on ensemble outputs.
• pRR uses partial matching (token/semantic overlap) rather than strict exact match, so minor formatting differences still score points.
• Transfer learning from SOQUAL only improved AraELECTRA-discriminator; applying it to other models did not yield gains.

Evidence (verbatim from paper)

As advised by the task organisers, we used partial Reciprocal Rank (pRR) score to measure the model performance. It is a variant of the traditional Reciprocal Rank evaluation metric that considers partial matching. We also report Exact Match (EM), and F1@1 in the results tables, which are evaluation metrics applied only to the top predicted answer. The EM metric is a binary measure that rewards a system only if the top predicted answer matches exactly one of the gold answers. In comparison, the F1@1 metric measures the token overlap between the top predicted answer and the best matching gold answer.

Citation

@misc{premasiri2022dtw,
  title={DTW at Qur'an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domain},
  author={Premasiri et al. (2022)},
  year={2022},
  note={arXiv:2205.06025}
}

• arXiv: 2205.06025