Glue Low Resource Eval

glue-low-resource-eval

Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts — Sai Ashish Somayajula et al. (2024) (arXiv:2403.12918, 2024)

What this evaluates

Evaluates the generalization and stability of finetuned pretrained language models on low-resource NLP tasks. It probes how well models adapt to sentiment classification, natural language inference, paraphrasing, similarity assessment, and linguistic acceptability when trained on severely limited data (300–1000 examples).

Datasets

• GLUE — total ?; splits: train (-1), dev (-1); HF glue

Metrics

• averaged evaluation metrics (primary) — range: percent
  • Mean of task-specific metrics (accuracy or F1 for classification, Pearson correlation for STS-B) across the eight GLUE datasets (SST-2, RTE, QNLI, MNLI, MRPC, STS-B, QQP, CoLA). Reported as mean ± standard deviation over 10 random seeds.

Input / output format

Input: Text pairs or single sentences depending on the GLUE task (e.g., sentence pairs for NLI/paraphrasing, single sentences for sentiment/acceptability).

Output: Predicted class labels or regression scores for each instance.

Scoring recipe

metrics = []
for task in GLUE_TASKS:
    preds = model.predict(task.dev_set)
    gold = task.dev_labels
    metrics.append(compute_task_metric(preds, gold)) # accuracy/F1/Pearson
glue_avg = sum(metrics) / len(metrics)
return glue_avg
# Repeat over 10 random seeds for low-resource splits (300/500/1000/1K)
# Report mean and std of glue_avg across seeds

Common pitfalls

• Low-resource splits are created by randomly downsampling the training set to 300, 500, 1000, or 1000 examples per seed, which can introduce high variance.
• Vanilla finetuning on BERT_LARGE often suffers from degenerate seeds on RTE due to vanishing gradients, inflating standard deviation.
• The BLO framework splits the training data into two disjoint subsets: one for optimizing task weights and another for optimizing attention parameters, which differs from standard single-split finetuning.

Evidence (verbatim from paper)

We finetune the models on the training set for each of the mentioned datasets and evaluated its performance on the original development set using the checkpoint obtained at the end of training... The averaged results over ten random seeds are reported in the paper. Table 1 summarizes our results. We compare our method with Vanilla, CHILD-TUNING_D, and DPS dense method using BERT_LARGE across 300, 500, and 1000 training data splits. Reported results are the averaged evaluation metrics over all eight GLUE datasets for each training data split.

Citation

@misc{somayajula2024generalizable,
  title={Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts},
  author={Sai Ashish Somayajula et al. (2024)},
  year={2024},
  note={arXiv:2403.12918}
}

• arXiv: 2403.12918