German Text Embedding Clustering Eval

german-text-embedding-clustering-eval

German Text Embedding Clustering Benchmark — Wehrli et al. (2024) (arXiv:2401.02709, 2024)

What this evaluates

Evaluates the quality of text embeddings for German-language documents by measuring how well they cluster into predefined topical categories. It probes a model's ability to capture semantic similarity and domain-specific nuances across different text lengths (titles vs. full texts) and sources.

Datasets

• BlurbsClusteringS2S/P2P — total ?; splits: test (-1); repo https://github.com/ClimSocAna/tecb-de
• TenKGnadClusteringS2S/P2P — total ?; splits: test (-1); repo https://github.com/ClimSocAna/tecb-de
• SubredditClusteringS2S/P2P — total ?; splits: test (-1); repo https://github.com/ClimSocAna/tecb-de

Metrics

• V-measure (primary) — range: [0, 1]
  • Measures clustering quality by combining homogeneity (each cluster contains only members of a single class) and completeness (all members of a given class are assigned to the same cluster). Outputs a score between 0 and 1, with 1 indicating perfect alignment with ground-truth topical categories.

Input / output format

Input: Raw text strings (titles, blurbs, news articles, or Reddit submissions) provided for embedding generation.

Output: Cluster assignments for each text instance, compared against ground-truth topical categories.

Scoring recipe

def compute_v_measure(true_labels, pred_labels):
    homogeneity = compute_homogeneity(true_labels, pred_labels)
    completeness = compute_completeness(true_labels, pred_labels)
    if homogeneity + completeness == 0:
        return 0.0
    return 2 * (homogeneity * completeness) / (homogeneity + completeness)

Common pitfalls

• The benchmark uses 10 random subsamples (splits) of varying sizes (10-100% of available data) per dataset, so reported scores can fluctuate significantly depending on the specific split drawn.
• V-measure evaluates clustering against ground-truth topical categories but does not inherently account for hierarchical granularity differences; researchers must explicitly configure splits for coarse vs. fine-grained genre clustering.
• German datasets are smaller and less standardized than English counterparts, requiring careful handling of data privacy constraints (e.g., providing submission IDs instead of raw text for Reddit) and domain coverage.

Evidence (verbatim from paper)

The evaluation is based on the V-measure Rosenberg and Hirschberg (2007). Given a ground truth, the V-measure outputs a score between 0 and 1, measuring homogeneity (clusters contain only one class) and completeness (clusters contain all class samples). MTEB uses topical categories derived from the data, such as the scientific discipline of a publication or newsgroup, as the ground truth.

Citation

@misc{wehrli2024german,
  title={German Text Embedding Clustering Benchmark},
  author={Wehrli et al. (2024)},
  year={2024},
  note={arXiv:2401.02709}
}

• arXiv: 2401.02709