Japanese Sts Ir Eval

japanese-sts-ir-eval

Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generation — Handa et al. (2025) (arXiv:2503.09094, 2025)

What this evaluates

Evaluates Japanese sentence embeddings on domain-specific semantic textual similarity (STS) and information retrieval (IR) tasks. It probes the model's ability to capture fine-grained semantic similarity in clinical text and retrieve relevant question-answer pairs in an educational domain.

Datasets

• JACSTS — total 3670; splits: test (3670); repo https://github.com/sociocom/Japanese-Clinical-STS
• QABot — total 1142; splits: test (1142)

Metrics

• Spearman's rank correlation (primary) — range: other
  • Measures the correlation between the ranking of predicted cosine similarity scores and the ranking of human-annotated similarity scores (0-5).
• MAP — range: [0, 1]
  • Mean Average Precision across all queries. Computes the average precision for each query's ranked list of questions and averages these scores across all 20 queries.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank across all queries. Computes the reciprocal rank of the first relevant question for each query and averages these values across all 20 queries.
• P@N — range: [0, 1]
  • Precision at cutoff N. Computes the proportion of relevant questions found in the top N retrieved results for each query, then averages across queries.

Input / output format

Input: Sentence pairs (for STS) or a query paired with a set of candidate questions (for IR).

Output: Cosine similarity score between sentence embeddings (for STS) or a ranked list of candidate questions based on cosine similarity (for IR).

Scoring recipe

def score_sts(pairs, embed):
    preds = [cosine_sim(embed(s1), embed(s2)) for s1, s2 in pairs]
    gold = [score for _, score in pairs]
    return spearmanr(preds, gold).correlation

def score_ir(queries, embed):
    maps, mrrs, pats = [], [], []
    for q, questions, labels in queries:
        scores = [cosine_sim(embed(q), embed(qst)) for qst in questions]
        ranked = argsort(scores, descending=True)
        rel = [labels[i] for i in ranked]
        maps.append(average_precision(rel))
        mrrs.append(1.0/(rel.index(1)+1) if 1 in rel else 0.0)
        pats.append(sum(rel[:N])/N)
    return mean(maps), mean(mrrs), mean(pats)

Common pitfalls

• Using Pearson correlation instead of Spearman's rank correlation for STS, as the protocol explicitly requires rank-based evaluation.
• Failing to specify the cutoff N when reporting P@N, since the metric is strictly cutoff-dependent and must be defined per experiment.
• Averaging IR metrics per question instead of per query, which contradicts the paper's definition of MAP/MRR across the 20 queries.

Evidence (verbatim from paper)

The evaluation is performed using Spearman’s rank correlation that measures the correlation between the ranking of predicted similarity scores and and the one of human-annotated scores. The performance is evaluated using a Mean Average Precision (MAP), Mean Reciprocal Rank (MRR) and Precision at N (P@N). The MAP represents the mean of average precision scores across all queries to measure the overall ranking quality by considering both precision and recall. The MRR indicates the average of reciprocal ranks of the first relevant question to each of 20 queries, and represents how quickly relevant questions are retrieved. P@N implies the proportion of relevant questions found in the top N retrieved ones to reflect the precision at a specific cutoff point.

Citation

@misc{handa2025domain,
  title={Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generation},
  author={Handa et al. (2025)},
  year={2025},
  note={arXiv:2503.09094}
}

• arXiv: 2503.09094