Csr L Eval

csr-l-eval

Code-Switching Information Retrieval: Benchmarks, Analysis, and the Limits of Current Retrievers — Zeng et al. (2026) (arXiv:2604.17632, 2026)

What this evaluates

Evaluates the robustness of information retrieval models when processing code-switched queries (English mixed with Mandarin Chinese or Japanese). It probes whether multilingual retrievers and rerankers suffer embedding divergence or performance degradation compared to monolingual English queries across argument, code, biomedical, and instruction-following retrieval tasks.

Datasets

• Touché 2020 — total 49; splits: test (49)
• HumanEval — total 158; splits: test (158)
• TRECCOVID — total 50; splits: test (50)
• FollowIR — total 198; splits: test (198)

Metrics

• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computes the weighted sum of relevance scores for the top 10 retrieved documents, normalized by the ideal DCG@10 for the same query to yield a score between 0 and 1.
• pairwise-MRR (p-MRR) — range: [0, 1]
  • Pairwise Mean Reciprocal Rank used specifically for the FollowIR instruction-following task. Measures the average reciprocal rank of the first relevant document in pairwise comparisons.

Input / output format

Input: A code-switched query (English mixed with Mandarin Chinese or Japanese) and a target document corpus. For cross-encoder models, the input is explicitly formatted as query-document pairs.

Output: A ranked list of documents (top-10) or a relevance score for each query-document pair.

Scoring recipe

def compute_ndcg_at_k(retrieved_docs, relevant_docs, k=10):
    dcg = 0.0
    for i, doc in enumerate(retrieved_docs[:k]):
        rel = 1 if doc in relevant_docs else 0
        dcg += rel / math.log2(i + 2)
    idcg = sum(1 / math.log2(i + 2) for i in range(min(len(relevant_docs), k)))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Interpreting cross-encoder scores as two-stage reranking results instead of direct full-corpus scoring, which inflates or misrepresents performance.
• Assuming automated metrics can reliably judge code-switching naturalness; the benchmark explicitly relies on human-annotated rewrites to ensure data quality.
• Applying nDCG@10 to the FollowIR subset, which explicitly requires pairwise-MRR (p-MRR) as the evaluation metric.

Evidence (verbatim from paper)

We use nDCG@10 as the primary metric throughout the evaluation, with the exception of FollowIR, where we report pairwise-MRR (p-MRR). For each method, we compare performance on the original queries and their code-switched counterparts. For the cross-encoder results in CSR-L, we score each query–document pair directly over the full document set, rather than reranking a top-$k$ candidate pool produced by a separate first-stage retriever.

Citation

@misc{zeng2026codeswitching,
  title={Code-Switching Information Retrieval: Benchmarks, Analysis, and the Limits of Current Retrievers},
  author={Zeng et al. (2026)},
  year={2026},
  note={arXiv:2604.17632}
}

• arXiv: 2604.17632