Metaicl Fewshot Eval

metaicl-fewshot-eval

Few-shot Adaptation Works with UnpredicTable Data — Chan et al. (2022) (arXiv:2208.01009, 2022)

What this evaluates

Evaluates the few-shot adaptation capability of language models on a diverse set of NLP tasks. It measures how well a model generalizes to unseen tasks after being fine-tuned on automatically extracted few-shot examples from web tables.

Datasets

• Min et al. (2021) Tasks — total 142; splits: test (52), train (90), dev (50)
• CROSSFIT — total ?; splits: (unstated)
• UNIFIEDQA — total ?; splits: (unstated)

Metrics

• mean Dev Tasks score (primary) — range: [0, 1]
  • Average of task-specific scores (e.g., accuracy or exact match) across all tasks in the split. Tasks are evaluated independently using their standard NLP metric, then averaged.

Input / output format

Input: Prompt containing few-shot demonstrations (input-output pairs) from the training data, followed by a test instance to be predicted.

Output: Model-generated prediction for the test instance (e.g., class label, text span, or numerical answer).

Scoring recipe

def compute_mean_score(predictions, golds, task_metrics):
    task_scores = []
    for pred, gold, metric in zip(predictions, golds, task_metrics):
        task_scores.append(metric(pred, gold))
    return sum(task_scores) / len(task_scores)

Common pitfalls

• Using the Dev set for final reporting instead of the strictly held-out Test set.
• Averaging raw token probabilities or log-likelihoods instead of task-specific metrics (e.g., accuracy) before computing the mean.
• Training on the Test Tasks, which are explicitly excluded from the Train set to prevent data leakage.

Evidence (verbatim from paper)

We make use of 3 different task splits: Test Tasks (52 tasks) The union of all test tasks from the 7 task settings in Min et al. (2021). Train Tasks (90 tasks) Contains all tasks in Min et al. (2021) except those which are Test Tasks. Dev Tasks (50 tasks) Contains all our Train Tasks except those which are not multiple-choice. We train for 5 epochs and evaluate after each epoch, selecting the checkpoint with the highest mean Dev Tasks score. We report scores of the selected checkpoint evaluated on the Test Tasks.

Citation

@misc{chan2022fewshotadaptation,
  title={Few-shot Adaptation Works with UnpredicTable Data},
  author={Chan et al. (2022)},
  year={2022},
  note={arXiv:2208.01009}
}

• arXiv: 2208.01009