Soi Id Ood Accuracy Eval

soi-id-ood-accuracy-eval

SOI Matters: Analyzing Multi-Setting Training Dynamics in Pretrained Language Models via Subsets of Interest — Vassef et al. (2025) (arXiv:2507.15236, 2025)

What this evaluates

Evaluates pretrained language models' in-distribution (ID) and out-of-distribution (OOD) classification accuracy under single-setting and multi-setting fine-tuning configurations. It probes how training dynamics and subset selection affect robustness and generalization across languages, sources, and tasks.

Datasets

• SST-2 — total ?; splits: train (-1), test (-1)
• IMDB — total ?; splits: train (-1), test (-1)
• Yelp — total ?; splits: train (-1), test (-1)
• Sentiment140 — total ?; splits: train (-1), test (-1)
• RTE — total ?; splits: train (-1), test (-1)
• QQP — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly predicted labels out of the total number of instances in the evaluation set.

Input / output format

Input: Single text sequence for sentiment tasks, or text pairs (premise-hypothesis for entailment, sentence pairs for paraphrase).

Output: Discrete class label (e.g., positive/negative for sentiment, entailment/neutral/contradiction for NLI, duplicate/non-duplicate for paraphrase).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• OOD evaluation targets differ by setting: translated Burmese for multi-lingual, SST-2/RTE/QQP for multi-source/multi-task.
• Second-stage fine-tuning is evaluated exclusively on OOD sets, not ID.
• Multi-setting gains must be compared against single-setting baselines to isolate the effect of joint training.

Evidence (verbatim from paper)

For Sentiment140, we observed the most significant gain, with a 7% improvement in OOD accuracy.

Citation

@misc{vassef2025soi,
  title={SOI Matters: Analyzing Multi-Setting Training Dynamics in Pretrained Language Models via Subsets of Interest},
  author={Vassef et al. (2025)},
  year={2025},
  note={arXiv:2507.15236}
}

• arXiv: 2507.15236