Pteb Eval

pteb-eval

PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs — Frank et al. (2025) (arXiv:2510.06730, 2025)

What this evaluates

Evaluates the robustness of sentence embedding models to token-level variations by measuring performance degradation when test instances are stochastically paraphrased at evaluation time. It probes whether models maintain semantic invariance under LLM-generated paraphrases that preserve meaning but alter surface form.

Datasets

• MTEB (STS & Non-STS tasks) — total ?; splits: test (-1)

Metrics

• Spearman’s rank correlation (primary) — range: [0, 1]
  • Non-parametric measure of rank correlation between predicted similarity scores (cosine similarity of embeddings) and gold human ratings. Computed as ρ = 1 - (6Σd_i²)/(n(n²-1)) for tied ranks, or via standard rank correlation function.

Input / output format

Input: Sentence pairs (for STS, pair classification, reranking, retrieval) or single texts/documents (for classification, clustering, summarization). For PTEB, each instance is dynamically paraphrased into 6 variants using an LLM (gemma3-27b) before encoding.

Output: Fixed-dimensional embedding vectors for each text. For STS, cosine similarity between the two embeddings is computed and compared to a gold rating on a 0–5 scale.

Scoring recipe

def score_pteb(embeddings, gold_ratings, n_runs=6):
    spearman_scores = []
    for run in range(n_runs):
        sim_scores = cosine_similarity(embeddings[run])
        rho = spearmanr(sim_scores, gold_ratings).correlation
        spearman_scores.append(rho)
    return np.mean(spearman_scores) * 100  # percentage

# Compute Δ = Original_Score - PTEB_Score
# Validate significance with Wilcoxon signed-rank test + Holm correction

Common pitfalls

• Failing to use sufficient paraphrase samples (n=6 is recommended) leads to high variance in standard deviation estimates and unreliable Δ calculations.
• Ignoring statistical significance testing; performance drops must be validated with Wilcoxon signed-rank tests and Holm-corrected p-values rather than relying on raw Δ differences.
• Assuming paraphrases perfectly preserve semantics without LLM judge or human verification, which can introduce semantic drift and confound robustness measurements.

Evidence (verbatim from paper)

Table 2: Spearman’s rank correlation between LLM Judge similarity scores and gold ratings for STS datasets; best scores bold. (in %)

Citation

@misc{frank2025pteb,
  title={PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMs},
  author={Frank et al. (2025)},
  year={2025},
  note={arXiv:2510.06730}
}

• arXiv: 2510.06730