Ms Marco Eval

ms-marco-eval

MS MARCO: A Human Generated MAchine Reading COmprehension Dataset — Bajaj et al. (2016) (arXiv:1611.09268, 2016)

What this evaluates

Evaluates machine reading comprehension models on real-world search queries across multiple answer types (numeric, yes/no, descriptive) and tasks (answer generation, span extraction, passage ranking). Probes a model's ability to extract or generate accurate answers from noisy, multi-document web contexts and handle unanswerable questions.

Datasets

• MS MARCO — total ?; splits: train (-1), test (-1)

Metrics

• ROUGE-L (primary) — range: [0, 1]
  • Longest common subsequence overlap between predicted and reference answers, normalized by reference length. Used as the headline metric for descriptive and generative QA tasks.
• BLEU — range: [0, 1]
  • Geometric mean of modified n-gram precisions (typically up to 4-grams) with a brevity penalty. Applied to subsets with multiple human references.
• pa-BLEU — range: [0, 1]
  • Pairwise BLEU that computes similarity between a prediction and multiple human references, then aggregates scores to measure consensus across diverse phrasings.
• Accuracy — range: [0, 1]
  • Exact match rate for numeric and yes/no answer categories.

Input / output format

Input: Question paired with one or more candidate web passages (context). For cloze-style tasks, a text sequence with a masked entity.

Output: Generated natural language answer, predicted answer span from the passage, or a confidence score for the selected span.

Scoring recipe

def compute_rouge_l(pred, refs):
    best_ref = max(refs, key=lambda r: lcs_length(pred, r))
    lcs_len = lcs_length(pred, best_ref)
    return lcs_len / len(best_ref.split())

def compute_accuracy(pred, gold):
    return 1.0 if normalize(pred) == normalize(gold) else 0.0

def compute_bleu(pred, refs):
    return standard_bleu_score(refs, pred)

Common pitfalls

• Using a single reference answer for descriptive questions instead of the curated multi-reference set, which underestimates model performance and ignores natural phrasing diversity.
• Applying exact-match accuracy to open-ended descriptive answers where paraphrasing is expected and ROUGE-L should be used instead.
• Ignoring the unanswerable question subset in the novice task, leading to inflated scores on models that always predict an answer.

Evidence (verbatim from paper)

We use accuracy and precision-recall measures for numeric answers and apply metrics like ROUGE-L (Lin, 2004) and phrasing-aware evaluation framework (Mitra et al., 2016) for long textual answers. The evaluation requires several reference answers per question that are each curated by a different human editor, thus providing a natural way to estimate how diversely a group of individuals may phrase the answer to the same question.

Citation

@misc{bajaj2016msmarco,
  title={MS MARCO: A Human Generated MAchine Reading COmprehension Dataset},
  author={Bajaj et al. (2016)},
  year={2016},
  note={arXiv:1611.09268}
}

• arXiv: 1611.09268