Glue Robustness Eval

glue-robustness-eval

Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition — Joshi et al. (2026) (arXiv:2601.07239, 2026)

What this evaluates

This protocol evaluates whether language models memorize benchmark surface features or demonstrate true semantic robustness. It measures performance degradation when inputs are paraphrased, lexically/syntactically perturbed, or adversarially rewritten, contrasting deterministic greedy decoding with stochastic distributional evaluation.

Datasets

• GLUE (MNLI, QQP, QNLI, SST-2) — total ?; splits: orig (-1), para (-1), pert (-1), adv (-1); HF glue

Metrics

• GLUE robustness ratio (primary) — range: [0, 1]
  • R_t^(d)(m) = (A_{t,para}^{(d)}(m) + A_{t,pert}^{(d)}(m) + A_{t,adv}^{(d)}(m)) / (3 * A_{t,orig}^{(d)}(m)), where A is the accuracy under decoding mode d on the respective dataset variant. Values range from 0 to 1 when variant performance does not exceed original performance.

Input / output format

Input: Natural-language task description with constrained answer format (e.g., options A/B/C), applied to original, paraphrased, perturbed, or adversarially rewritten GLUE examples.

Output: Single text completion per prompt. Deterministic mode yields one completion; stochastic mode yields K=10 independent completions. A label-extraction function phi maps each completion to a task label.

Scoring recipe

for each task t, model m, decoding mode d in [Det, Stoch]:
  for each variant v in [orig, para, pert, adv]:
    generate completions for D_t^v
    extract labels phi(y_hat) for each completion
    if d == Stoch: majority_label = mode(phi(y_hat_1)...phi(y_hat_K))
    else: majority_label = phi(y_hat_det)
    A_t_v = mean(majority_label == gold_label)
R = (A_orig_para + A_orig_pert + A_orig_adv) / (3 * A_orig_orig)
return R

Common pitfalls

• Reporting only deterministic accuracy hides distributional generalization; stochastic majority-vote often yields higher robustness scores.
• The robustness ratio assumes variant performance does not exceed original performance; if it does, R can exceed 1, complicating interpretation.
• Label extraction phi must be consistent across all variants; inconsistent parsing invalidates accuracy comparisons.

Evidence (verbatim from paper)

To isolate robustness rather than absolute accuracy, we define a GLUE robustness ratio for each triplet $(t,m,d)$: R_t^(d)(m) = (A_{t,para}^{(d)}(m) + A_{t,pert}^{(d)}(m) + A_{t,adv}^{(d)}(m)) / (3 A_{t,orig}^{(d)}(m)). By construction, R_t^(d)(m) \in [0,1] whenever the model performs no better on the variants than on the original split.

Citation

@misc{joshi2026stochasticchaos,
  title={Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition},
  author={Joshi et al. (2026)},
  year={2026},
  note={arXiv:2601.07239}
}

• arXiv: 2601.07239