Decanlp Eval

decanlp-eval

The Natural Language Decathlon: Multitask Learning as Question Answering — McCann et al. (2018) (arXiv:1806.08730, 2018)

What this evaluates

This benchmark evaluates a model's ability to perform multitask learning across ten diverse natural language processing tasks by framing them as a unified question-answering problem. It probes zero-shot generalization, domain adaptation, and the effectiveness of anti-curriculum training strategies without relying on task-specific modules.

Datasets

• decaNLP — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• decaScore (primary) — range: score
  • Composite benchmark score aggregating performance across the 10 diverse NLP tasks. Exact aggregation details are provided in Appendix D.

Input / output format

Input: Question and context provided as input sequences. For baselines, they are concatenated into a single sequence; for the MQAN model, they are processed as separate sequences with a coattention mechanism.

Output: Generated answer text, produced via a multi-pointer-generator decoder that dynamically selects between generating tokens from an external vocabulary, pointing to the question, or pointing to the context.

Scoring recipe

def compute_score(predictions, gold, task_type):
    if task_type in ['SST', 'MNLI', 'MWSC', 'SNLI', 'Amazon', 'Yelp']:
        return 1.0 if predictions == gold else 0.0
    elif task_type in ['SQuAD', 'QA-SRL']:
        return nF1_score(predictions, gold)
    elif task_type == 'WikiSQL':
        return lfem_score(predictions, gold)
    # decaScore aggregates these task-specific scores across all 10 tasks

Common pitfalls

• Assuming the model uses task-specific heads or modules; the paper explicitly evaluates a unified architecture without task-specific parameters.
• Misinterpreting the input format for baselines vs. MQAN; baselines concatenate question and context, while MQAN uses separate sequences with coattention, which significantly affects the model's ability to copy tokens.
• Overlooking that decaScore is a composite metric; evaluating on a single task does not reflect the full benchmark performance.

Evidence (verbatim from paper)

In Table 2, validation metrics reveal that the S2S model does not perform well on SQuAD. On WikiSQL, it obtains a much higher score than prior sequence-to-sequence baselines [Zhong et al., 2017], but it is low compared to MQAN (+QPtr) and the other baselines. Augmenting the S2S model with self-attentive (w/ SAtt) encoder and decoder layers Vaswani et al. [2017], as detailed in C, increases the model's capacity to integrate information from both context and question. This improves performance on SQuAD by 20 nF1, QA-SRL by 4 nF1, and WikiSQL by 12 LFEM.

Citation

@misc{mccann2018decanlp,
  title={The Natural Language Decathlon: Multitask Learning as Question Answering},
  author={McCann et al. (2018)},
  year={2018},
  note={arXiv:1806.08730}
}

• arXiv: 1806.08730