Clues Fewshot Eval

clues-fewshot-eval

CLUES: Few-Shot Learning Evaluation in Natural Language Understanding — Mukherjee et al. (2021) (arXiv:2111.02570, 2021)

What this evaluates

Evaluates few-shot learning capabilities of pre-trained language models across sentence classification, question answering, and named entity recognition tasks. It measures how well models adapt with limited labeled examples (10, 20, 30 shots) compared to fully supervised settings and human performance.

Datasets

• SST-2 — total ?; splits: train (-1), test (-1)
• MNLI — total ?; splits: train (-1), test (-1)
• NER — total ?; splits: train (-1), test (-1)
• MRC — total ?; splits: train (-1), test (-1)

Metrics

• macro-averaged results (primary) — range: percent
  • Standard classification accuracy averaged across all classes/tasks. For span-extraction tasks (NER, MRC), the unified formulation predicts start and end token positions, but results are reported as macro-averaged accuracy percentages.

Input / output format

Input: For classification: prepended special token (e.g., [CLS]) followed by task text. For NER/MRC: unified span-extraction format. Prompt-based methods use pattern templates with verbalizers. In-context learning prepends k labeled demonstrations.

Output: Classification: predicted label. NER/MRC: predicted span start and end token positions (i, j).

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
macro_accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• Prompt-based fine-tuning works well for classification but yields near-random performance on NER/MRC without task-specific adaptation.
• GPT-3 in-context learning succeeds on simple tasks (SST-2) but fails on complex ones (MNLI, NER, MRC), contradicting the assumption that larger models always scale better in few-shot settings.
• Model size does not consistently improve few-shot performance for classic fine-tuning, and larger models can exhibit higher variance across training splits.

Evidence (verbatim from paper)

For each task, we report macro-averaged results for each model trained on five different splits and evaluated on the corresponding test split along with the standard deviation.

Citation

@misc{mukherjee2021clues,
  title={CLUES: Few-Shot Learning Evaluation in Natural Language Understanding},
  author={Mukherjee et al. (2021)},
  year={2021},
  note={arXiv:2111.02570}
}

• arXiv: 2111.02570