Uquad1.0 Eval

uquad1.0-eval

UQuAD1.0: Development of an Urdu Question Answering Training Data for Machine Reading Comprehension — Kazi et al. (2021) (arXiv:2111.01543, 2021)

What this evaluates

This benchmark evaluates Machine Reading Comprehension (MRC) capabilities in Urdu by testing a model's ability to extract correct answer spans from context paragraphs in response to questions. It probes span prediction accuracy, handling of multiple valid answers, and performance across different question types and named entities.

Datasets

• UQuAD1.0 — total 49000; splits: test (-1)

Metrics

• Exact Match (EM) — range: [0, 1]
  • Returns 1 if the predicted answer string exactly matches the gold answer string, otherwise 0. It is a strict metric that penalizes partial matches or minor formatting differences.
• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall based on word-level overlap. Precision is the proportion of words in the prediction that appear in the gold answer. Recall is the proportion of words in the gold answer that appear in the prediction.

Input / output format

Input: A context paragraph and a corresponding question in Urdu.

Output: A predicted answer text span extracted from the context paragraph (derived from the model's predicted start and end indices).

Scoring recipe

def compute_metrics(predictions, golds):
    em_scores = []
    f1_scores = []
    for pred, gold_list in zip(predictions, golds):
        best_em = 0
        best_f1 = 0
        for gold in gold_list:
            if pred.strip().lower() == gold.strip().lower():
                best_em = 1
                best_f1 = 1.0
                break
            pred_words = set(pred.split())
            gold_words = set(gold.split())
            if not pred_words or not gold_words:
                continue
            precision = len(pred_words & gold_words) / len(pred_words)
            recall = len(pred_words & gold_words) / len(gold_words)
            f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
            best_f1 = max(best_f1, f1)
        em_scores.append(best_em)
        f1_scores.append(best_f1)
    return {'EM': sum(em_scores)/len(em_scores), 'F1': sum(f1_scores)/len(f1_scores)}

Common pitfalls

• Exact Match is extremely strict; answers with minor formatting differences, extra words, or slight paraphrasing score 0 despite high semantic overlap.
• The test set contains questions with multiple valid answers (23% of cases), so evaluators must check predictions against all possible gold answers rather than just the first one.
• F1 score is order-agnostic and based on word sets, so it can be high even if the predicted answer has words in a different order or includes extra context words.

Evidence (verbatim from paper)

We assess the performance of the three models on the UQuAD test set... The evaluation is carried out using two widely used measures for Machine Reading Comprehension: Accuracy/Exact Match (EM) metric and F1 score. The F1 score indicates the average overlap between the predicted response and the true answer, whereas the EM represents the proportion of predicted answers precisely matching accurate answers.

Citation

@misc{kazi2021uquad1.0,
  title={UQuAD1.0: Development of an Urdu Question Answering Training Data for Machine Reading Comprehension},
  author={Kazi et al. (2021)},
  year={2021},
  note={arXiv:2111.01543}
}

• arXiv: 2111.01543