Muld Eval

muld-eval

MuLD: The Multitask Long Document Benchmark — Hudson et al. (2022) (arXiv:2202.07362, 2022)

What this evaluates

Evaluates models' ability to process and extract information from long documents (minimum 10,000 tokens) across multiple NLP tasks including question answering, summarization, classification, and translation. It specifically probes long-context dependency handling and real-world document understanding capabilities.

Datasets

• MuLD Benchmark — total ?; splits: test (-1)
• NarrativeQA — total ?; splits: (unstated)
• HotpotQA — total ?; splits: (unstated)
• OpenSubtitles — total ?; splits: (unstated)

Metrics

• results (primary) — range: other
  • Task-specific evaluation metrics are used per subtask (e.g., QA, summarization, classification, translation). Exact formulas are not specified in the provided text but are standard for each task type.

Input / output format

Input: Long documents with a minimum length of 10,000 tokens, accompanied by task-specific prompts or questions.

Output: Task-dependent responses such as generated answers, summaries, class labels, or translated text.

Scoring recipe

# Pseudo-code for task-specific scoring
if task == 'QA':
    score = exact_match_or_f1(predictions, gold_answers)
elif task == 'summarization':
    score = rouge_l(predictions, gold_summaries)
elif task == 'classification':
    score = accuracy(predictions, gold_labels)
elif task == 'translation':
    score = bleu(predictions, gold_translations)
return score

Common pitfalls

• Models may fail if they cannot attend to the full 10,000+ token context window, leading to degraded performance on long-range dependencies.
• Task complexity varies significantly; for example, narrative understanding in NarrativeQA is harder than factual QA in HotpotQA due to longer context and multi-hop reasoning requirements.
• Translation tasks may suffer if the output sequence length is constrained (e.g., 512 tokens) regardless of input length.

Evidence (verbatim from paper)

The results for each of the benchmark tasks are presented in Table 2 for both the T5 and Longformer models. The Longformer model consistently outperforms the T5 model across many of the tasks, suggesting that models which are able to make use of a longer context perform well on our benchmark.

Citation

@misc{hudson2022muld,
  title={MuLD: The Multitask Long Document Benchmark},
  author={Hudson et al. (2022)},
  year={2022},
  note={arXiv:2202.07362}
}

• arXiv: 2202.07362