Text Clustering Eval

text-clustering-eval

Summaries as Centroids for Interpretable and Scalable Text Clustering — Diaz-Rodriguez et al. (2025) (arXiv:2502.09667, 2025)

What this evaluates

Evaluates the ability of centroid-based clustering algorithms to group unlabeled text documents into semantically coherent clusters. It measures clustering accuracy, label alignment with ground truth, and how closely learned centroids match true cluster centers.

Datasets

• Bank77 — total ?; splits: test (-1)
• CLINC — total ?; splits: test (-1)
• GoEmo — total ?; splits: test (-1)
• MASSIVE — total ?; splits: test (-1)
• StackExchange — total ?; splits: 2020 (69147), 2021 (54322), 2022 (43521), 2023 (38953)

Metrics

• ACC (primary) — range: percent
  • Accuracy of cluster assignments mapped to ground-truth labels via optimal permutation. Reported as a percentage.
• NMI (primary) — range: percent
  • Normalized Mutual Information measuring the mutual dependence between predicted clusters and ground-truth labels, normalized by the average entropy of both distributions. Reported as a percentage.
• dist — range: other
  • Average Euclidean distance between the algorithm's learned centroids and the ground-truth centroids across all clusters.

Input / output format

Input: Unlabeled text documents or queries to be clustered.

Output: Cluster assignment label for each input document.

Scoring recipe

def compute_metrics(predictions, gold, centroids_pred, centroids_true):
    acc = max(permutation_accuracy(predictions, gold)) * 100
    nmi = normalized_mutual_information(predictions, gold) * 100
    dist = np.mean([np.linalg.norm(c_p - c_t) for c_p, c_t in zip(centroids_pred, centroids_true)])
    return acc, nmi, dist

Common pitfalls

• Uses ground-truth number of clusters (k) for all methods, which is unrealistic for standard unsupervised settings.
• Embedding model choice significantly impacts ACC/NMI scores; results are not directly comparable across different embedding backbones.
• LLM-based variants use a fixed budget of prompts per iteration, independent of dataset size, unlike other LLM clustering methods.

Evidence (verbatim from paper)

Table 1 reports mean accuracy (ACC) and normalized mutual information (NMI) for multiple k-NLPmeans and k-LLMmeans variants... The dist metric directly gauges how closely each algorithm recovers the true centroids, an especially meaningful criterion for centroid-based methods.

Citation

@misc{diazrodriguez2025summaries,
  title={Summaries as Centroids for Interpretable and Scalable Text Clustering},
  author={Diaz-Rodriguez et al. (2025)},
  year={2025},
  note={arXiv:2502.09667}
}

• arXiv: 2502.09667