# Text Classification Comparison Eval

> Systematic comparison of generative (AR, MLM, Diffusion) and discriminative (encoder) transformer models on text classification tasks, focusing on sample efficiency, robustness to input noise, and output calibration/ordinality. Use when the user wants to benchmark on AG News, Emotion, SST2, SST5, Multiclass Sentiment Analysis, Twitter Financial News Sentiment, IMDb, Hate Speech Offensive, or asks about evaluating this task. Reports weighted-F1 score.

- Skill: `qhjqhj00/text-classification-comparison-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/text-classification-comparison-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/text-classification-comparison-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/text-classification-comparison-eval

---


# text-classification-comparison-eval

> Generative or Discriminative? Revisiting Text Classification in the Era of Transformers — Kasa et al. (2025) (arXiv:2506.12181, 2025)

## What this evaluates

Systematic comparison of generative (AR, MLM, Diffusion) and discriminative (encoder) transformer models on text classification tasks, focusing on sample efficiency, robustness to input noise, and output calibration/ordinality.

## Datasets

- **AG News** — total ?; splits: test (-1)
- **Emotion** — total ?; splits: test (-1)
- **SST2** — total ?; splits: test (-1)
- **SST5** — total ?; splits: test (-1)
- **Multiclass Sentiment Analysis** — total ?; splits: test (-1)
- **Twitter Financial News Sentiment** — total ?; splits: test (-1)
- **IMDb** — total ?; splits: test (-1)
- **Hate Speech Offensive** — total ?; splits: test (-1)

## Metrics

- `weighted-F1 score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall, weighted by the number of true instances for each class. Computed as the sum of per-class F1 scores multiplied by class support, divided by total support.
- `ECE` — range: [0, 1]
  - Expected Calibration Error measures the expected difference between predicted confidence and actual accuracy across confidence bins, weighted by bin size.
- `MCE` — range: [0, 1]
  - Maximum Calibration Error reports the worst-case calibration gap (absolute difference between confidence and accuracy) across all confidence bins.
- `MSE` — range: other
  - Mean Squared Error between predicted class indices and true class indices for ordinal classification tasks.
- `MAE` — range: other
  - Mean Absolute Error between predicted class indices and true class indices for ordinal classification tasks.
- `Unimodality (UM)` — range: [0, 1]
  - Metric verifying that the predicted probability distribution has a single peak, ensuring the model does not assign high confidence to distant ordinal categories simultaneously.

## Input / output format

**Input**: Text sequences (single sentences to paragraph-length passages) for classification. Optionally perturbed via random token drop (X% of tokens removed) or random token substitution (X% replaced with random vocabulary tokens, excluding special tokens).

**Output**: Predicted probability distribution over all target classes.

## Scoring recipe

```python
def compute_metrics(pred_probs, true_labels, num_classes):
    preds = np.argmax(pred_probs, axis=1)
    # Weighted F1
    f1 = f1_score(true_labels, preds, average='weighted')
    # Calibration (ECE/MCE)
    bins = np.linspace(0, 1, 10)
    ece, mce = 0.0, 0.0
    for i in range(len(bins)-1):
        mask = (pred_probs.max(axis=1) >= bins[i]) & (pred_probs.max(axis=1) < bins[i+1])
        if mask.sum() > 0:
            acc = (preds[mask] == true_labels[mask]).mean()
            conf = pred_probs[mask].max(axis=1).mean()
            ece += mask.sum() * abs(acc - conf)
            mce = max(mce, abs(acc - conf))
    # Ordinal (MSE/MAE/UM)
    mse = np.mean((preds - true_labels)**2)
    mae = np.mean(np.abs(preds - true_labels))
    um = 1.0 if np.argmax(pred_probs) == np.argmax(np.diff(pred_probs, prepend=0, append=0)) else 0.0
    return {'weighted-F1': f1, 'ECE': ece, 'MCE': mce, 'MSE': mse, 'MAE': mae, 'UM': um}
```

## Common pitfalls

- Evaluations are conducted across multiple training sample sizes (128 to full data) rather than a single fixed test split, requiring careful aggregation across data regimes.
- Noise robustness tests use unspecified perturbation percentages (X%) for token drop/substitution, which must be explicitly defined to reproduce results.
- Calibration and ordinal metrics require full probability distributions; using hard class predictions (argmax) will yield undefined or incorrect scores.

## Evidence (verbatim from paper)

> Performance is measured using the weighted-F1 score. ... For ordinal evaluation, we report MSE (Mean Squared Error), MAE (Mean Absolute Error), and Unimodality (UM). For calibration, we measure ECE (Expected Calibration Error) and MCE (Maximum Calibration Error).

## Citation

```bibtex
@misc{kasa2025generative,
  title={Generative or Discriminative? Revisiting Text Classification in the Era of Transformers},
  author={Kasa et al. (2025)},
  year={2025},
  note={arXiv:2506.12181}
}
```

- arXiv: 2506.12181