Cmedteb Eval

cmedteb-eval

CMedTEB & CARE: Benchmarking and Enabling Efficient Chinese Medical Retrieval via Asymmetric Encoders — Jiang et al. (2026) (arXiv:2604.10937, 2026)

What this evaluates

Evaluates Chinese medical text embedding models across retrieval, reranking, and semantic textual similarity tasks. It probes the model's ability to capture domain-specific semantic alignment while measuring the trade-off between retrieval accuracy and inference efficiency.

Datasets

• CMedTEB — total ?; splits: CMed v1 (-1), CMed v2 (-1), Retrieval (-1), Rerank (-1), STS (-1); repo https://github.com/PhilipGAQ/CARE

Metrics

• MAP@10 — range: [0, 1]
  • Mean Average Precision at 10. Computes the average precision of retrieved documents up to rank 10, averaged over all queries.
• nDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at 10. Measures ranking quality by discounting gains logarithmically based on position, normalized by the ideal ranking.
• Pearson — range: [-1, 1]
  • Pearson correlation coefficient measuring linear correlation between predicted similarity scores and ground-truth human ratings.
• Avg (primary) — range: [0, 1]
  • Arithmetic mean of MAP@10 (CMed v1), MAP@10 (CMed v2), nDCG@10 (Retrieval), MAP@10 (Rerank), and Pearson (STS).

Input / output format

Input: Query-document pairs for embedding generation; queries with candidate document sets for retrieval/reranking; sentence pairs for STS.

Output: Dense embedding vectors for queries and documents; ranked lists of document IDs/scores for retrieval/reranking; similarity scores for STS.

Scoring recipe

def map_at_10(retrieved, relevant):
    hits = 0; scores = []
    for i, doc in enumerate(retrieved[:10]):
        if doc in relevant:
            hits += 1
            scores.append(hits / (i + 1))
    return sum(scores) / len(relevant) if relevant else 0

def ndcg_at_10(retrieved, relevant):
    dcg = sum(1 / log2(i + 2) for i, doc in enumerate(retrieved[:10]) if doc in relevant)
    ideal = sum(1 / log2(i + 2) for i in range(min(len(relevant), 10)))
    return dcg / ideal if ideal > 0 else 0

pearson = scipy.stats.pearsonr(pred_scores, gold_scores)[0]
avg = (map_v1 + map_v2 + ndcg + map_rerank + pearson) / 5

Common pitfalls

• Confusing total model parameters with online inference parameters; the benchmark emphasizes that asymmetric models only count the lightweight query encoder for latency/cost.
• Averaging metrics across different tasks (retrieval, reranking, STS) without noting that STS uses Pearson correlation (0-1 range) while MAP/nDCG are percentages, which can skew the 'Avg' interpretation.
• Assuming symmetric baselines are directly comparable in latency; the paper explicitly notes that symmetric giants incur prohibitive computational costs despite higher accuracy.

Evidence (verbatim from paper)

Table 2 presents the evaluation of CARE series on the CMedTEB benchmark, alongside strong open-source baselines. We observe two key findings: (1) CARE establishes a new state of the art: the 0.3B-4B variant achieves an average score of 78.13, and the 0.3B-8B variant reaches 78.94, surpassing the strongest baseline gte-Qwen2-1.5B-instruct (77.61, a decoder-only model), despite using a much smaller query encoder.

Citation

@misc{jiang2026cmedteb,
  title={CMedTEB & CARE: Benchmarking and Enabling Efficient Chinese Medical Retrieval via Asymmetric Encoders},
  author={Jiang et al. (2026)},
  year={2026},
  note={arXiv:2604.10937}
}

• arXiv: 2604.10937