Uit Viquad 2.0 Eval

uit-viquad-2.0-eval

Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension — Son Quoc Tran et al. (2023) (arXiv:2303.13355, 2023)

What this evaluates

Evaluates Vietnamese language models' machine reading comprehension capabilities, specifically probing their ability to extract correct answer spans and correctly identify when a question cannot be answered from the given context.

Datasets

• UIT-ViQuAD 2.0 — total ?; splits: train (28457), dev (-1)

Metrics

• Exact Match (EM) (primary) — range: [0, 1]
  • The percentage of predicted answers that exactly match any one of the gold answer(s) annotated by human readers.
• F1-score — range: [0, 1]
  • The average token overlap between predicted and gold answers. For each question, the F1 score is calculated against each gold answer, and the maximum F1 is taken as the question's score.
• Recall_unanswerable — range: [0, 1]
  • The percentage of unanswerable questions that the model correctly predicts as not having an answer in the given context.
• Recall_answerable — range: [0, 1]
  • The percentage of answerable questions that the model attempts to answer, regardless of whether the predicted answer is correct.

Input / output format

Input: A Vietnamese context paragraph and a corresponding question.

Output: A predicted answer span (string) for span-extraction evaluation, or a binary classification (answerable/unanswerable) for recall metrics.

Scoring recipe

def compute_metrics(predictions, golds, is_answerable):
    em = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    f1 = sum(max(f1_score(p, g) for g in golds) for p in predictions) / len(predictions)
    un_mask = [not a for a in is_answerable]
    recall_un = sum(1 for p, ans in zip(predictions, un_mask) if p == '' and ans) / sum(un_mask)
    ans_mask = is_answerable
    recall_ans = sum(1 for p, ans in zip(predictions, ans_mask) if p != '' and ans) / sum(ans_mask)
    return {'EM': em, 'F1': f1, 'Recall_unanswerable': recall_un, 'Recall_answerable': recall_ans}

Common pitfalls

• Confusing span-extraction metrics (EM/F1) with classification metrics (Recall_unanswerable/Recall_answerable), as the paper evaluates both capabilities on the same dataset.
• The 'Monolingual hard' and 'Multilingual hard' question definitions are specific analytical constructs in this paper, not standard benchmark metrics.
• F1 calculation takes the maximum overlap across multiple gold answers per question, which differs from simple token-level averaging.

Evidence (verbatim from paper)

Following previous works (Rajpurkar et al., 2016, 2018; Nguyen et al., 2020a), we use two metrics, Exact Match (EM) and F1-score, to evaluate the overall performances of different models on Reading Comprehension task. ... Because we carry out our analysis on the test set that requires models having abilities to recognize unanswerable questions, we also take into consideration the performances of models in classifying answerable and unanswerable questions. Performances on classification tasks are reported in our analysis as Recall on answerable questions and unanswerable questions.

Citation

@misc{tran2023revealing,
  title={Revealing Weaknesses of Vietnamese Language Models Through Unanswerable Questions in Machine Reading Comprehension},
  author={Son Quoc Tran et al. (2023)},
  year={2023},
  note={arXiv:2303.13355}
}

• arXiv: 2303.13355