Raft Few Shot Classification Eval

raft-few-shot-classification-eval

RAFT: A Real-World Few-Shot Text Classification Benchmark — Alex et al. (2021) (arXiv:2109.14076, 2021)

What this evaluates

Evaluates few-shot text classification on real-world tasks with class imbalance and long inputs. It probes a model's ability to leverage limited labeled examples, domain knowledge, and open-domain retrieval to classify text without a validation set.

Datasets

• RAFT — total ?; splits: train (50), test (-1)

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, averaged across all classes (macro-averaged) for each dataset, then averaged across all 11 datasets.

Input / output format

Input: Natural language text input, task-specific instructions, and exactly 50 labeled few-shot examples. Models may optionally use open-domain web retrieval and unlabeled test data.

Output: A single predicted class label from the provided natural language class names.

Scoring recipe

def compute_raft_score(predictions, gold, datasets):
    dataset_f1s = []
    for ds in datasets:
        preds = predictions[ds]
        golds = gold[ds]
        # Compute macro-F1 per dataset
        f1 = macro_f1_score(golds, preds)
        dataset_f1s.append(f1)
    return sum(dataset_f1s) / len(dataset_f1s)

Common pitfalls

• Using a validation set for tuning is strictly forbidden; only 50 training examples are provided with no validation split.
• Assuming balanced class distributions; datasets exhibit heavy imbalance, making accuracy misleading and requiring macro-F1.
• Counting the 50-shot limit per class instead of per task; the limit applies to the entire task regardless of number of classes.

Evidence (verbatim from paper)

Since some RAFT datasets have substantial class imbalances, we use F1 as our evaluation metric. We compute macro-averaged F1 scores, even for binary datasets. To get an overall score, we average across all datasets.

Citation

@misc{alex2021raft,
  title={RAFT: A Real-World Few-Shot Text Classification Benchmark},
  author={Alex et al. (2021)},
  year={2021},
  note={arXiv:2109.14076}
}

• arXiv: 2109.14076