Glue Wikitext Eval

glue-wikitext-eval

Computational Economics in Large Language Models: Exploring Model Behavior and Incentive Design under Resource Constraints — Sandeep Reddy et al. (arXiv:2508.10426, 2025)

What this evaluates

Evaluates general language understanding across classification, regression, and entailment tasks, alongside language modeling capability. It also measures computational efficiency and internal attention allocation strategies under resource constraints.

Datasets

• GLUE Benchmark — total ?; splits: train (-1), dev (-1), test (-1); HF glue
• WikiText-103 — total ?; splits: train (-1), validation (-1), test (-1); HF wikitext

Metrics

• MNLI-m Accuracy (primary) — range: [0, 1]
  • Ratio of correctly predicted labels to total predictions on the MNLI mismatched split.
• STS-B Pearson/Spearman Correlation — range: [-1, 1]
  • Pearson and Spearman rank correlation coefficients between predicted and human similarity scores.
• CoLA MCC — range: [-1, 1]
  • Matthews Correlation Coefficient for binary grammatical acceptability classification.
• WikiText-103 Perplexity — range: [0, inf)
  • Exponential of the average negative log-likelihood of the test tokens: exp(-1/N * sum(log p(x_i))).
• FLOPS — range: other
  • Theoretical floating-point operations required for inference, hardware-independent measure of complexity.
• Inference Latency — range: other
  • Average wall-clock time in milliseconds to process a single sample on one A100 GPU (batch size 1).
• Gini Coefficient — range: [0, 1]
  • Measures inequality or concentration of attention weights across tokens.
• Shannon Entropy — range: [0, inf)
  • Measures uncertainty in attention distributions: -sum(p * log(p)).

Input / output format

Input: Tokenized sentence pairs or single sentences for GLUE tasks; tokenized text sequences for WikiText-103.

Output: Class labels or regression scores for GLUE; next-token probability distributions for WikiText-103; attention weight matrices for economic metrics.

Scoring recipe

def compute_metrics(predictions, gold, attention_weights):
    # Task metrics
    mnli_acc = np.mean(predictions['mnli'] == gold['mnli'])
    sts_corr = pearsonr(predictions['sts'], gold['sts'])[0]
    cola_mcc = matthews_corrcoef(gold['cola'], predictions['cola'])
    wikitext_ppl = np.exp(-np.mean(np.log(predictions['wikitext'])))
    # Efficiency & Economic metrics
    flops = compute_flops(model_config)
    latency = measure_wallclock_time(model, batch_size=1)
    gini = compute_gini(attention_weights)
    entropy = -np.sum(attention_weights * np.log(attention_weights + 1e-9))
    # Average economic metrics across layers, heads, and test set
    gini = np.mean(gini)
    entropy = np.mean(entropy)
    return {'mnli_acc': mnli_acc, 'sts_corr': sts_corr, 'cola_mcc': cola_mcc,
            'wikitext_ppl': wikitext_ppl, 'flops': flops, 'latency': latency,
            'gini': gini, 'entropy': entropy}

Common pitfalls

• Using the MNLI matched split instead of the mismatched split for evaluation.
• Averaging Gini and Shannon entropy per-instance before aggregating across layers, heads, and the test set, rather than averaging attention weights first.
• Failing to apply early stopping based on validation set performance during fine-tuning, which can lead to overfitting and inflated test scores.

Evidence (verbatim from paper)

We use the standard evaluation metric for each respective dataset: MNLI-m (Accuracy), STS-B (Pearson/Spearman correlation), CoLA (Matthews Correlation Coefficient), and WikiText-103 (Perplexity). Our evaluation is designed to be comprehensive, capturing not only the final task performance but also the computational efficiency and the internal strategic behavior of the models.

Citation

@misc{reddy2025computationaleconomics,
  title={Computational Economics in Large Language Models: Exploring Model Behavior and Incentive Design under Resource Constraints},
  author={Sandeep Reddy et al.},
  year={2025},
  note={arXiv:2508.10426}
}

• arXiv: 2508.10426