Self Adaptive Curriculum Nlu Eval

self-adaptive-curriculum-nlu-eval

Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding — Qi Feng et al. (2025) (arXiv:2507.09758, 2025)

What this evaluates

Evaluates whether self-adaptive curriculum learning strategies, which use pre-trained model confidence to estimate example difficulty, improve fine-tuning performance over random and length-based sampling baselines across multiple NLU tasks.

Datasets

• SST-2 — total ?; splits: train (-1), val (-1), test (-1)
• SST-5 — total ?; splits: train (-1), val (-1), test (-1)
• HSOL — total ?; splits: train (-1), val (-1), test (-1)
• XNLI — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted labels out of the total number of instances.

Input / output format

Input: Single sentences (SST-2, SST-5, HSOL) or sentence pairs (XNLI), tokenized for classification.

Output: Predicted class label from the task-specific label set (e.g., positive/negative, 5 sentiment classes, hate/offensive/neither, entailment/neutral/contradiction).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Dataset class imbalance is explicitly noted for HSOL, which can skew accuracy without weighted metrics.
• Difficulty scoring relies on the pre-trained model's initial confidence, which varies by architecture (BERT vs RoBERTa) and may not correlate perfectly with human difficulty.
• Few-shot vs full-dataset regimes are evaluated separately, so results are not directly comparable across data scales.

Evidence (verbatim from paper)

We evaluate our proposed methods on four publicly available datasets, covering diverse NLP tasks to demonstrate the generality of our approach. ... SST-2*(Socher et al., [2013])* is a balanced binary sentiment analysis dataset containing movie review sentences labeled as positive or negative. ... The Hate Speech Offensive Language dataset*(Davidson et al., [2017])* includes tweets labeled into three categories: hate speech, offensive language, and neither, with a significant class imbalance.

Citation

@misc{feng2025selfadaptive,
  title={Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding},
  author={Qi Feng et al. (2025)},
  year={2025},
  note={arXiv:2507.09758}
}

• arXiv: 2507.09758