I Star Eval

i-star-eval

Stable Anisotropic Regularization — Rudman et al. (2023) (arXiv:2305.19358, 2023)

What this evaluates

This evaluation probes how anisotropic regularization (I-STAR) affects the downstream performance of fine-tuned language models across standard NLP benchmarks. It also measures the geometric properties of the resulting embedding spaces, specifically isotropy and intrinsic dimensionality, to correlate representation structure with task accuracy.

Datasets

• SST-2 — total ?; splits: test (-1)
• QNLI — total ?; splits: test (-1)
• RTE — total ?; splits: test (-1)
• MRPC — total ?; splits: test (-1)
• QQP — total ?; splits: test (-1)
• COLA — total ?; splits: test (-1)
• STS-B — total ?; splits: test (-1)
• SST-5 — total ?; splits: test (-1)
• SQUAD — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted class labels out of total instances.
• Matthew's Correlation — range: [0, 1]
  • Correlation coefficient between predicted and true binary/multiclass labels, accounting for class imbalance.
• Pearson's Correlation — range: [0, 1]
  • Linear correlation coefficient between predicted continuous scores and ground truth values.
• F1/EM — range: percent
  • Exact Match (EM) measures exact span overlap; F1 measures token-level overlap between predicted and gold spans.

Input / output format

Input: Tokenized text sequences or sentence pairs, optionally with task-specific tokens, fed into pre-trained transformer models (ALBERT, BERT, DistilBERT) for fine-tuning.

Output: Class labels for classification tasks; extracted text spans for SQUAD.

Scoring recipe

def compute_metrics(predictions, gold, task):
    if task == 'COLA': return matthews_corrcoef(gold, predictions)
    if task == 'STS-B': return pearsonr(gold, predictions)
    if task == 'SQUAD': return exact_match_f1(gold, predictions), exact_match(gold, predictions)
    return accuracy_score(gold, predictions)

Common pitfalls

• Performance gains from I-STAR are task- and model-dependent; high noise in tasks like MRPC and COLA can obscure isotropy-performance correlations.
• Metrics are averaged over only 5 random seeds, which may not fully capture fine-tuning variance or stability.
• CosReg is often misinterpreted as improving isotropy, but the evaluation shows it only shifts activation means without increasing isotropy.

Evidence (verbatim from paper)

For COLA, we report Matthew's Correlation; for STS-B, we report Pearson's Correlation; for SQUAD, we present F1/EM. For all remaining tasks, we report accuracy. We report the average/standard deviation over 5 random seeds.

Citation

@misc{rudman2023stableanisotropic,
  title={Stable Anisotropic Regularization},
  author={Rudman et al. (2023)},
  year={2023},
  note={arXiv:2305.19358}
}

• arXiv: 2305.19358