Code Retrieval Eval

code-retrieval-eval

CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking — Tarun Suresh et al. (2024) (arXiv:2412.01007, 2024)

What this evaluates

Evaluates the ability of code embedding models and LLM-based rerankers to retrieve relevant code snippets or functions given natural language queries. It probes semantic matching between text descriptions (e.g., GitHub issues, function comments) and code across multiple programming languages, as well as function localization in real-world software repositories.

Datasets

• CodeSearchNet — total ?; splits: test (-1)
• AdvTest — total ?; splits: test (-1)
• CoIR — total ?; splits: test (-1)
• SWE-Bench-Lite — total 274; splits: test (274)

Metrics

• MRR@1000 (primary) — range: percent
  • Mean Reciprocal Rank at cutoff 1000. For each query, compute 1/rank of the first relevant document, then average across all queries.
• nDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff 10. Measures ranking quality by discounting gains logarithmically based on position, normalized by the ideal DCG.
• MRR@100 — range: percent
  • Mean Reciprocal Rank at cutoff 100, used for evaluating the listwise reranker on top-100 retrieved results.
• Top-K Accuracy — range: percent
  • Percentage of queries where the ground-truth function/file appears within the top K retrieved results (K=1,2,3,5,10).

Input / output format

Input: Natural language query (e.g., function description or GitHub issue) for retrieval; for reranking, the query plus the top-100 retrieved code snippets.

Output: A ranked list of code snippets or functions ordered by relevance to the query.

Scoring recipe

def compute_mrr_at_k(ranks, k):
    relevant_ranks = [r for r in ranks if r <= k]
    if not relevant_ranks:
        return 0.0
    return 1.0 / min(relevant_ranks)

def compute_ndcg_at_k(scores, labels, k):
    ranked = sorted(zip(scores, labels), reverse=True)[:k]
    dcg = sum((2**l - 1) / math.log2(i + 2) for i, (_, l) in enumerate(ranked))
    ideal = sorted(labels, reverse=True)[:k]
    idcg = sum((2**l - 1) / math.log2(i + 2) for i, l in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Zero-shot evaluation: models are not fine-tuned on the test benchmarks, so performance reflects generalization rather than dataset-specific adaptation.
• SWE-Bench-Lite localization only considers examples where patches modify existing functions/classes; examples introducing new functions or imports are excluded.
• CoIR aggregates results across multiple task types (code-to-text, code-to-code, hybrid), which may require careful averaging to match reported numbers.

Evidence (verbatim from paper)

We report the official metrics for each dataset: MRR@1000 for CodeSearchNet and Advtest, and nDCG@10 for COIR. During inference, the top 100 results from our code retriever are passed to the reranker, with evaluation conducted using MRR@100.

Citation

@misc{suresh2024cornstack,
  title={CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking},
  author={Tarun Suresh et al. (2024)},
  year={2024},
  note={arXiv:2412.01007}
}

• arXiv: 2412.01007