Mteb Negation Eval

mteb-negation-eval

Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Models — Günther et al. (2023) (arXiv:2307.11224, 2023)

What this evaluates

Evaluates the semantic similarity and retrieval capabilities of sentence embedding models across diverse downstream tasks. It also probes the models' sensitivity to grammatical negation and their ability to distinguish syntactically similar negative examples from entailments.

Datasets

• MTEB benchmark — total ?; splits: test (-1)
• Negation dataset — total ?; splits: test (-1); HF jinaai/negation-dataset

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, measuring the quality of ranked retrieval results against ground truth relevance labels.
• Spearman — range: other
  • Spearman rank correlation coefficient between predicted cosine similarities and human-annotated similarity scores.
• EasyNegation — range: percent
  • Percentage of triplets where the model ranks the anchor-entailment pair closer than the anchor-negative pair.
• HardNegation — range: percent
  • Percentage of triplets where the model ranks the anchor-entailment pair closer than the entailment-negative pair.

Input / output format

Input: Sentence pairs for similarity tasks; query-document pairs for retrieval; triplets (anchor, entailment, negative) for negation evaluation.

Output: Fixed-dimensional embedding vectors (e.g., 384, 512, 768, or 1024 dimensions) for each input text.

Scoring recipe

# Compute cosine similarities
sim_qd = cosine(query_emb, doc_embs)
sim_an = cosine(anchor_emb, entail_emb)
sim_ane = cosine(anchor_emb, neg_emb)
sim_ene = cosine(entail_emb, neg_emb)

# Retrieval (nDCG@10)
ndcg = compute_ndcg_at_k(sim_qd, k=10)

# STS (Spearman)
spearman_corr = spearmanr(predicted_sims, gold_sims).correlation

# Negation
easy_neg = mean(sim_an > sim_ane)
hard_neg = mean(sim_an > sim_ene)

Common pitfalls

• MTEB aggregates scores across many tasks; averaging them without weighting can mask performance on specific domains like biomedical or legal retrieval.
• Negation metrics (Easy/Hard) rely on cosine similarity thresholds; models trained primarily on similarity tasks may struggle with HardNegation due to syntactic similarity between entailment and negative pairs.
• Embedding dimensions vary across models (384 to 1024); comparing raw scores without normalizing or accounting for dimensionality can lead to unfair comparisons.

Evidence (verbatim from paper)

We measure performance with respect to two metrics: one measures the percentage of samples where the model positions the anchor and entailment closer than the anchor and negative (which is an easy task, as the anchor and negative are syntactically dissimilar), the other measures the percentage of samples where the model positions the anchor and entailment closer than the entailment and negative (which is a hard task, as the entailment and negative are syntactically more similar than the anchor and entailment). The former is denoted by EasyNegation, the latter by HardNegation.

Citation

@misc{gunther2023jinaembeddings,
  title={Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Models},
  author={Günther et al. (2023)},
  year={2023},
  note={arXiv:2307.11224}
}

• arXiv: 2307.11224