Embedding Clustering Eval

embedding-clustering-eval

Evaluating Embedding Generalization: How LLMs, LoRA, and SLERP Shape Representational Geometry — Kabane (2025) (arXiv:2511.21703, 2025)

What this evaluates

This evaluation probes how well different embedding models capture underlying number-theoretic structures in numeric sequences. It measures the quality of the latent representation space by comparing clustering performance against ground-truth mathematical group labels versus unsupervised KMeans assignments.

Datasets

• number-theoretic-sequences — total ?; splits: test (-1)

Metrics

• Silhouette Coefficient (primary) — range: [-1, 1]
  • Quantifies how similar an object is to its own cluster compared to other clusters. Higher values indicate better-defined, more separated clusters.
• Davies–Bouldin Index — range: other
  • Computes the average similarity ratio of each cluster with its most similar one, based on scatter and separation measures. Lower values indicate better clustering quality.

Input / output format

Input: Numeric sequences with associated number-theoretic group labels (for supervised evaluation) or unlabeled numeric sequences (for unsupervised KMeans evaluation).

Output: Embedding vectors for each sequence, followed by clustering assignments or direct metric computation against true labels.

Scoring recipe

embeddings = model.encode(sequences)
sil_sup = silhouette_score(embeddings, true_labels)
db_sup = davies_bouldin_score(embeddings, true_labels)
kmeans_labels = KMeans(n_clusters=k).fit_predict(embeddings)
sil_unsup = silhouette_score(embeddings, kmeans_labels)
db_unsup = davies_bouldin_score(embeddings, kmeans_labels)
return sil_sup, db_sup, sil_unsup, db_unsup

Common pitfalls

• Negative Silhouette scores indicate poor global structure, but local regularities may still exist and be captured by KMeans.
• A higher KMeans Silhouette score than the true-label score suggests the embedding space clusters according to learned geometric priors rather than mathematical definitions.

Evidence (verbatim from paper)

Two standard metrics are employed: the Silhouette Coefficient (higher is better) and the Davies–Bouldin Index (lower is better).

Citation

@misc{kabane2025evaluatingembedding,
  title={Evaluating Embedding Generalization: How LLMs, LoRA, and SLERP Shape Representational Geometry},
  author={Kabane (2025)},
  year={2025},
  note={arXiv:2511.21703}
}

• arXiv: 2511.21703