Vimrc Eval

vimrc-eval

VLSP 2021 - ViMRC Challenge: Vietnamese Machine Reading Comprehension — Nguyen et al. (2022) (arXiv:2203.11400, 2022)

What this evaluates

Evaluates Vietnamese machine reading comprehension models on span extraction from passages, including handling unanswerable questions. It measures how well systems can locate exact answer spans or correctly identify when no answer exists in the context.

Datasets

• UIT-ViQuAD 2.0 (ViMRC) — total ?; splits: public test (-1), private test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Token-level F1 score between predicted answer span and gold answer span. Calculated as the harmonic mean of precision and recall over word tokens, following SQuAD 2.0 conventions.
• Exact Match (EM) — range: [0, 1]
  • Binary metric indicating whether the predicted answer string exactly matches the gold answer string.

Input / output format

Input: A Vietnamese context passage and a corresponding question.

Output: A predicted answer span extracted from the passage, or a flag indicating the question is unanswerable.

Scoring recipe

def compute_metrics(predictions, golds):
    f1_scores = []
    em_scores = []
    for pred, gold in zip(predictions, golds):
        pred_tokens = set(pred.split())
        gold_tokens = set(gold.split())
        if not pred_tokens or not gold_tokens:
            f1 = 0.0
        else:
            common = pred_tokens & gold_tokens
            precision = len(common) / len(pred_tokens)
            recall = len(common) / len(gold_tokens)
            f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
        em = 1.0 if pred == gold else 0.0
        f1_scores.append(f1)
        em_scores.append(em)
    return sum(f1_scores)/len(f1_scores), sum(em_scores)/len(em_scores)

Common pitfalls

• Models must correctly predict unanswerable questions; predicting a span when none exists heavily penalizes F1.
• Evaluation uses token-level F1, not character-level, following SQuAD 2.0 conventions.
• Human baseline uses majority voting over 4 annotators per question, which differs from standard single-annotator benchmarks.

Evidence (verbatim from paper)

Following the answering phase, we compute the human accuracy by F1-score and exact match scores for both public and private tests. The ranking results of the team are based on F1 points for both rounds.

Citation

@misc{nguyen2022vimrc,
  title={VLSP 2021 - ViMRC Challenge: Vietnamese Machine Reading Comprehension},
  author={Nguyen et al. (2022)},
  year={2022},
  note={arXiv:2203.11400}
}

• arXiv: 2203.11400