Text Classification Comparison Eval

text-classification-comparison-eval

Generative or Discriminative? Revisiting Text Classification in the Era of Transformers — Kasa et al. (2025) (arXiv:2506.12181, 2025)

What this evaluates

Systematic comparison of generative (AR, MLM, Diffusion) and discriminative (encoder) transformer models on text classification tasks, focusing on sample efficiency, robustness to input noise, and output calibration/ordinality.

Datasets

• AG News — total ?; splits: test (-1)
• Emotion — total ?; splits: test (-1)
• SST2 — total ?; splits: test (-1)
• SST5 — total ?; splits: test (-1)
• Multiclass Sentiment Analysis — total ?; splits: test (-1)
• Twitter Financial News Sentiment — total ?; splits: test (-1)
• IMDb — total ?; splits: test (-1)
• Hate Speech Offensive — total ?; splits: test (-1)

Metrics

• weighted-F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, weighted by the number of true instances for each class. Computed as the sum of per-class F1 scores multiplied by class support, divided by total support.
• ECE — range: [0, 1]
  • Expected Calibration Error measures the expected difference between predicted confidence and actual accuracy across confidence bins, weighted by bin size.
• MCE — range: [0, 1]
  • Maximum Calibration Error reports the worst-case calibration gap (absolute difference between confidence and accuracy) across all confidence bins.
• MSE — range: other
  • Mean Squared Error between predicted class indices and true class indices for ordinal classification tasks.
• MAE — range: other
  • Mean Absolute Error between predicted class indices and true class indices for ordinal classification tasks.
• Unimodality (UM) — range: [0, 1]
  • Metric verifying that the predicted probability distribution has a single peak, ensuring the model does not assign high confidence to distant ordinal categories simultaneously.

Input / output format

Input: Text sequences (single sentences to paragraph-length passages) for classification. Optionally perturbed via random token drop (X% of tokens removed) or random token substitution (X% replaced with random vocabulary tokens, excluding special tokens).

Output: Predicted probability distribution over all target classes.

Scoring recipe

def compute_metrics(pred_probs, true_labels, num_classes):
    preds = np.argmax(pred_probs, axis=1)
    # Weighted F1
    f1 = f1_score(true_labels, preds, average='weighted')
    # Calibration (ECE/MCE)
    bins = np.linspace(0, 1, 10)
    ece, mce = 0.0, 0.0
    for i in range(len(bins)-1):
        mask = (pred_probs.max(axis=1) >= bins[i]) & (pred_probs.max(axis=1) < bins[i+1])
        if mask.sum() > 0:
            acc = (preds[mask] == true_labels[mask]).mean()
            conf = pred_probs[mask].max(axis=1).mean()
            ece += mask.sum() * abs(acc - conf)
            mce = max(mce, abs(acc - conf))
    # Ordinal (MSE/MAE/UM)
    mse = np.mean((preds - true_labels)**2)
    mae = np.mean(np.abs(preds - true_labels))
    um = 1.0 if np.argmax(pred_probs) == np.argmax(np.diff(pred_probs, prepend=0, append=0)) else 0.0
    return {'weighted-F1': f1, 'ECE': ece, 'MCE': mce, 'MSE': mse, 'MAE': mae, 'UM': um}

Common pitfalls

• Evaluations are conducted across multiple training sample sizes (128 to full data) rather than a single fixed test split, requiring careful aggregation across data regimes.
• Noise robustness tests use unspecified perturbation percentages (X%) for token drop/substitution, which must be explicitly defined to reproduce results.
• Calibration and ordinal metrics require full probability distributions; using hard class predictions (argmax) will yield undefined or incorrect scores.

Evidence (verbatim from paper)

Performance is measured using the weighted-F1 score. ... For ordinal evaluation, we report MSE (Mean Squared Error), MAE (Mean Absolute Error), and Unimodality (UM). For calibration, we measure ECE (Expected Calibration Error) and MCE (Maximum Calibration Error).

Citation

@misc{kasa2025generative,
  title={Generative or Discriminative? Revisiting Text Classification in the Era of Transformers},
  author={Kasa et al. (2025)},
  year={2025},
  note={arXiv:2506.12181}
}

• arXiv: 2506.12181