Hyrec Cmteb Eval

hyrec-cmteb-eval

HyReC: Exploring Hybrid-based Retriever for Chinese — Wang et al. (2025) (arXiv:2506.21913, 2025)

What this evaluates

Evaluates the retrieval capability of hybrid (dense + sparse/lexicon) models on Chinese text. It probes how well the model ranks relevant passages for a given query across diverse Chinese domains like medical, e-commerce, and general web.

Datasets

• C-MTEB — total ?; splits: T2Retrieval (-1), MMarcoRetrieval (-1), DuRetrieval (-1), CovidRetrieval (-1), CmedqaRetrieval (-1), EcomRetrieval (-1), MedicalRetrieval (-1), VideoRetrieval (-1)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. It measures the quality of the ranked list of retrieved documents by comparing the discounted gain of relevant documents against the ideal sorted list, normalized to [0,1].

Input / output format

Input: Query string and a list of candidate passage strings (or a corpus to search over).

Output: Ranked list of passage IDs or strings, typically truncated to top-10 for evaluation.

Scoring recipe

def compute_ndcg_at_10(relevant_docs, retrieved_docs, k=10):
    dcg = 0.0
    for i, doc in enumerate(retrieved_docs[:k]):
        rel = 1.0 if doc in relevant_docs else 0.0
        dcg += rel / math.log2(i + 2)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_docs), k)))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• The benchmark aggregates scores across 8 diverse Chinese sub-tasks; reporting only the average without per-task breakdowns hides domain-specific weaknesses.
• The hybrid model combines dense and sparse scores using a Normalization Module (NM) scaled to [0,1]; failing to normalize scores before fusion leads to unstable training and poor retrieval performance.
• Chinese tokenization ambiguity affects sparse/lexicon components; using standard Jieba without the proposed semantic union alignment degrades performance.

Evidence (verbatim from paper)

Adhering to the official benchmark protocols, we evaluate our method using Pyserini and utilize $nDCG@10$ as the primary evaluation metric.

Citation

@misc{wang2025hyrec,
  title={HyReC: Exploring Hybrid-based Retriever for Chinese},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2506.21913}
}

• arXiv: 2506.21913