Negation Embedding Eval

negation-embedding-eval

Semantic Adapter for Universal Text Embeddings: Diagnosing and Mitigating Negation Blindness to Enhance Universality — Cao (2025) (arXiv:2504.00584, 2025)

What this evaluates

Probes whether text embedding models suffer from 'negation blindness' by testing their ability to correctly identify paraphrases over negated counterparts, and measures the trade-off with semantic similarity correlation.

Datasets

• STSB — total ?; splits: train (-1), test (-1); HF glue/stsb
• SemAntoNeg — total ?; splits: train (-1), test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly identified paraphrase candidates out of total test instances. For STSB, candidates are the positive paraphrase and its negated counterpart. For SemAntoNeg, candidates are three expressions with negation/antonym substitutions.
• Correlation — range: [-1, 1]
  • Pearson correlation coefficient between the model's computed embedding similarity scores and human-annotated semantic similarity values, following the standard STSB evaluation protocol.

Input / output format

Input: Sentence pairs (query, positive paraphrase) for STSB; query plus three candidate paraphrases with negation/antonym substitutions for SemAntoNeg. Models compute cosine similarity between query embeddings and candidate embeddings.

Output: Discrete class label (selected paraphrase candidate) for accuracy; continuous similarity score for correlation.

Scoring recipe

# Accuracy
correct = sum(1 for q, pos, neg in test
              if sim(q, pos) > sim(q, neg))
accuracy = correct / len(test)

# Correlation (STSB)
preds = [sim(s1, s2) for s1, s2 in test]
gold = [score for _, _, score in test]
correlation = pearsonr(preds, gold).statistic

Common pitfalls

• STSB test set merges official validation and test splits.
• SemAntoNeg requires 10-fold repeated random sampling (32/68 split) due to missing predefined splits.
• Adapter application typically reduces Pearson correlation by ~1% despite boosting accuracy.

Evidence (verbatim from paper)

In order to test the effectiveness of the proposed embedding re-weighting method, a paraphrase detection task is formulated using the STSB dataset: the Group 5 data from STSB [$sentence_1_i$, $sentence_2_i$] are used as paraphrase text pairs. The embedding models are then required to identify the accurate paraphrase of $sentence_1_i$ from [$sentence_2_i$, $neg_sentence_1_i$]. ... Two evaluation metrics are used in this experiment: Accuracy: measures if state-of-the-art text embeddings can correctly identify semantically similar text $sentence_2_i$ from negated text $neg_sentence_1_i$ which has similar surface form as $sentence_1_i$. Correlation: measures if the similarity score of state-of-the-art text embeddings correlates well with human annotated similarity values using Pearson correlation following the evaluation protocol of STSB.

Citation

@misc{cao2025semanticadapter,
  title={Semantic Adapter for Universal Text Embeddings: Diagnosing and Mitigating Negation Blindness to Enhance Universality},
  author={Cao (2025)},
  year={2025},
  note={arXiv:2504.00584}
}

• arXiv: 2504.00584