Geneol Eval

geneol-eval

GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings — Thirukovalluru et al. (2024) (arXiv:2410.14635, 2024)

What this evaluates

Evaluates training-free sentence embedding quality by aggregating LLM-generated semantic variations. Probes semantic similarity preservation and cross-task robustness without model fine-tuning.

Datasets

• STS benchmark — total ?; splits: test (-1)
• MTEB — total ?; splits: test (-1)

Metrics

• Spearman rank correlation (cosine similarity) (primary) — range: [-1, 1]
  • Spearman rank correlation between cosine similarities of aggregated sentence embeddings and human-provided similarity scores.

Input / output format

Input: Single sentences or sentence pairs. For each sentence, the method generates m semantically equivalent transformations using an LLM, then computes the mean of their embeddings.

Output: A single fixed-dimensional sentence embedding vector per input, obtained by averaging the embeddings of the original sentence and its m generated transformations.

Scoring recipe

preds = []
for s1, s2 in test_pairs:
    emb1 = mean(embed(s1), *[embed(t) for t in generate(s1, m)])
    emb2 = mean(embed(s2), *[embed(t) for t in generate(s2, m)])
    preds.append(cosine_similarity(emb1, emb2))
return spearman_corr(preds, gold_scores)

Common pitfalls

• Using training splits for STS tasks, which the authors explicitly exclude.
• Unequally distributing the generation budget (m) across sentence pairs in similarity tasks drastically reduces performance compared to equal allocation.
• Relying on the final hidden layer for embedding extraction; penultimate layers often yield better results for training-free methods.

Evidence (verbatim from paper)

We evaluate GenEOLand other baselines on the STS benchmark Conneau and Kiela ([2018]). Spearman rank correlation (cosine similarity) is the main metric Muennighoff et al. ([2022]). Training sets of the STS tasks are not used. We additionally asses GenEOLon 10 MTEB tasks across 4 categories (Classification, Clustering, Reranking and Pair Classification) using the appropriate metrics described in Muennighoff et al. ([2022]).

Citation

@misc{thirukovalluru2024geneol,
  title={GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings},
  author={Thirukovalluru et al. (2024)},
  year={2024},
  note={arXiv:2410.14635}
}

• arXiv: 2410.14635