Ftc Ensemble Eval

ftc-ensemble-eval

Ensembling Finetuned Language Models for Text Classification — Pineda Arango et al. (2024) (arXiv:2410.19889, 2024)

What this evaluates

Evaluates whether post-hoc ensembling strategies (e.g., greedy selection, top-N, model averaging) improve classification accuracy and uncertainty calibration over single fine-tuned language models. It probes the robustness of combining multiple finetuned classifiers across varying training data sizes (10% vs 100%).

Datasets

• DBpedia — total ?; splits: train (-1), val (-1), test (-1)
• News — total ?; splits: train (-1), val (-1), test (-1)
• SetFit — total ?; splits: train (-1), val (-1), test (-1)
• SST-2 — total ?; splits: train (-1), val (-1), test (-1)
• Tweet — total ?; splits: train (-1), val (-1), test (-1)
• IMDB — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• classification error (primary) — range: [0, 1]
  • 1 minus accuracy; the fraction of test instances where the ensemble's predicted class does not match the ground truth label.
• negative log-likelihood (NLL) — range: other
  • Mean negative log probability of the true class across the test set: -1/N * sum(log(p(y_i|x_i))). Used to assess uncertainty calibration and robustness.

Input / output format

Input: Text classification instances (input text and ground truth label) from six benchmark datasets, evaluated under two training data regimes (10% and 100% of the available training split).

Output: A final predicted class label and associated class probabilities for each test instance, derived by combining predictions from five fine-tuned language models using a specified ensemble strategy (e.g., greedy selection, top-N, model averaging).

Scoring recipe

def compute_metrics(ensemble_probs, gold_labels):
    preds = np.argmax(ensemble_probs, axis=1)
    error = 1.0 - np.mean(preds == gold_labels)
    nll = -np.mean(np.log(ensemble_probs[np.arange(len(gold_labels)), gold_labels] + 1e-12))
    return error, nll

Common pitfalls

• The ensemble selection/weighting is optimized on the validation split, but all reported metrics must be computed strictly on the held-out test split.
• Larger ensembles (N=50) do not consistently outperform smaller ones (N=5); performance gains are mixed for Top-50 and Random-50 strategies.
• Models are finetuned on either 10% or 100% of the training data, which drastically changes the base performance and ensemble dynamics.

Evidence (verbatim from paper)

We measure the negative log-likelihood (NLL) and the classification error on the test data, while we use the validation split for training the ensemble.

Citation

@misc{pinedaarango2024ensembling,
  title={Ensembling Finetuned Language Models for Text Classification},
  author={Pineda Arango et al. (2024)},
  year={2024},
  note={arXiv:2410.19889}
}

• arXiv: 2410.19889