Coir Eval

coir-eval

CoIR: A Comprehensive Benchmark for Code Information Retrieval Models — Xiangyang Li et al. (2024) (arXiv:2407.02883, 2024)

What this evaluates

Evaluates code information retrieval models across diverse tasks including text-to-code, code-to-code, code-to-text, and hybrid code retrieval. It probes a model's ability to handle semi-structured, syntactically complex code snippets and natural language queries across multiple programming languages and domains.

Datasets

• APPS — total 14000; splits: train (5000), test (3800)
• CosQA — total 20600; splits: test (-1)
• Synthetic Text2SQL — total 106000; splits: train (100000), test (6000)
• CodeSearchNet — total 1000000; splits: train (905000), dev (41000), test (53000)
• CodeSearchNet-CCR — total 1000000; splits: train (905000), dev (41000), test (53000)
• CodeTransOcean-DL — total ?; splits: test (-1)
• CodeTransOcean-Contest — total 1000; splits: train (561), dev (226), test (446)
• StackOverflow QA — total 20000; splits: train (13000), dev (3000), test (2000)
• CodeFeedQA — total ?; splits: test (-1)
• CodeFeedback-MT — total 66000; splits: train (53000), test (13000)

Metrics

• nDCG (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at a given cutoff (typically @10 or @100). Computed as DCG@k divided by IDCG@k, where DCG ranks relevant documents by their graded relevance scores discounted logarithmically by position.
• precision — range: [0, 1]
  • Fraction of retrieved documents that are relevant at a specific cutoff k.
• recall — range: [0, 1]
  • Fraction of all relevant documents that are successfully retrieved at cutoff k.
• MAP — range: [0, 1]
  • Mean Average Precision across all queries, averaging the precision values at each rank where a relevant document is retrieved.

Input / output format

Input: A query (natural language, code snippet, or mixed text/code) and a candidate corpus of code/text documents to retrieve from.

Output: A ranked list of corpus documents or relevance scores for each query, typically output as JSON.

Scoring recipe

def compute_ndcg(relevant_docs, predicted_ranking, k=10):
    dcg = 0.0
    for i, doc_id in enumerate(predicted_ranking[:k]):
        rel = 1 if doc_id in relevant_docs else 0
        dcg += rel / math.log2(i + 2)
    idcg = sum(1 / math.log2(i + 2) for i in range(min(len(relevant_docs), k)))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Models often overfit to specific benchmarks like CodeSearchNet, leading to inflated performance but poor generalization across diverse code domains.
• Multi-turn retrieval tasks require handling dialogue contexts exceeding 4,000 tokens, which exceeds the standard 512-token context window of many retrieval models.
• Code is semi-structured and syntactically complex, making standard text-based retrieval metrics insufficient without careful handling of code-specific tokenization and formatting.

Evidence (verbatim from paper)

Unlike traditional evaluations that require manual coding and result collection, CoIR offers an automated pipeline for both open-source and proprietary models, supporting metrics such as nDCG, precision, recall, and MAP. Results are stored in JSON format for easy access.

Citation

@misc{li2024coir,
  title={CoIR: A Comprehensive Benchmark for Code Information Retrieval Models},
  author={Xiangyang Li et al. (2024)},
  year={2024},
  note={arXiv:2407.02883}
}

• arXiv: 2407.02883