C Sts Eval

c-sts-eval

CASE -- Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurement — Zhang et al. (2025) (arXiv:2503.17279, 2025)

What this evaluates

Evaluates how well sentence embedding models capture conditional semantic similarity by measuring how accurately they rank similarity scores under different conditions compared to human ratings. It probes condition-aware representation learning and the ability to modulate embeddings based on specific contextual constraints.

Datasets

• C-STS — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Spearman Rank correlation (primary) — range: [-1, 1]
  • Computes Spearman’s rank correlation between the model's predicted cosine similarity scores and the re-annotated human similarity ratings on the test set.
• Accuracy — range: [0, 1]
  • For each sentence pair with two conditions c1, c2 and human ratings y1 > y2, computes cosine similarity under each condition. A prediction is correct if (sim_c1 - sim_c2)(y1 - y2) > 0. Accuracy is the fraction of correct predictions over N test instances.

Input / output format

Input: Pairs of sentences (s1, s2) and two conditions (c1, c2) with corresponding human similarity ratings (y1, y2).

Output: Condition-aware sentence embeddings CASE(s1, c) and CASE(s2, c), from which cosine similarity scores are computed.

Scoring recipe

def compute_metrics(predictions, gold, N):
    # predictions: dict of {condition: cosine_similarity}
    # gold: dict of {condition: human_rating}
    
    # Spearman Rank Correlation
    pred_scores = [predictions[c] for c in conditions]
    gold_scores = [gold[c] for c in conditions]
    spearman_corr = spearmanr(pred_scores, gold_scores).correlation
    
    # Accuracy
    correct = 0
    for pair in test_set:
        sim_c1 = cosine_similarity(embed(s1, c1), embed(s2, c1))
        sim_c2 = cosine_similarity(embed(s1, c2), embed(s2, c2))
        y1, y2 = gold[pair]['c1'], gold[pair]['c2']
        if (sim_c1 - sim_c2) * (y1 - y2) > 0:
            correct += 1
    accuracy = correct / N
    return spearman_corr, accuracy

Common pitfalls

• Using the original C-STS validation set instead of the re-annotated version, which contains ambiguous/invalid conditions that skew results.
• Confusing standard STS evaluation with C-STS; the task requires comparing similarity rankings under different conditions, not just absolute similarity scores.
• Forgetting to subtract the condition embedding (c) in post-processing, which significantly impacts embedding isotropy and final performance.

Evidence (verbatim from paper)

We evaluate the performance of sentence embedding models on two metrics: Spearman Rank correlation and Accuracy. We compute Spearman’s rank correlation between the similarity scores by CASE and the re-annotated human ratings on the test set. ... A prediction is considered correct if (sim_c1 - sim_c2)(y1 - y2) > 0, which evaluates whether the model’s predicted similarity ranking aligns with the human annotations. Then, the accuracy is given by [formula].

Citation

@misc{zhang2025case,
  title={CASE -- Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurement},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2503.17279}
}

• arXiv: 2503.17279