Cat Benchmark Eval

cat-benchmark-eval

Evaluating and Optimizing the Effectiveness of Neural Machine Translation in Supporting Code Retrieval Models: A Study on the CAT Benchmark — Hung Phan et al. (2023) (arXiv:2308.04693, 2023)

What this evaluates

Evaluates whether augmenting code search models with neural machine translation-generated AST representations improves retrieval accuracy over raw code tokens. Probes the capability of NMT to translate natural language queries into compact abstract syntax tree non-terminal sequences and measures the downstream impact on code retrieval performance.

Datasets

• TLC — total ?; splits: train (53592), val (7561), test (7584)
• CSN — total ?; splits: train (323225), val (8849), test (19317)
• Funcom — total ?; splits: train (1184437), val (20000), test (20000)
• PCSD — total ?; splits: train (57846), val (19000), test (19028)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: average of 1/rank for each query, where rank is the position of the first relevant code candidate in the retrieved list.
• CrystalBLEU-4 — range: [0, 1]
  • 4-gram based textual similarity metric with filtering of repetitive n-grams, specifically adapted for evaluating translated software artifacts.
• Meteor — range: [0, 1]
  • Alignment-based metric weighing word stems and synonyms, shown to correlate better with human judgment than BLEU for code translation tasks.
• Average_EffectMRR — range: percent
  • Average MRR improvement across two embedding dimensions (20, 768) and two base models (GraphCodeBERT, UniXcoder): sum(EffectMRR)/4.

Input / output format

Input: Natural language query paired with a candidate method declaration (code). For retrieval, the model receives query text and candidate code embeddings/vectors.

Output: Ranked list of candidate code snippets based on similarity scores, or a binary relevance judgment per candidate for rank calculation.

Scoring recipe

def compute_mrr(relevance_scores):
    ranks = [i+1 for i, s in enumerate(relevance_scores) if s == 1]
    return 1.0 / ranks[0] if ranks else 0.0

def compute_average_effect_mrr(test_queries, ast_model, base_models, dims):
    total_effect = 0.0
    count = 0
    for d in dims:
        for model in base_models:
            mrr_org = mean([compute_mrr(model.encode(query, dim=d)) for query in test_queries])
            mrr_com = mean([compute_mrr(ast_model.augment(query, dim=d)) for query in test_queries])
            total_effect += (mrr_com - mrr_org)
            count += 1
    return total_effect / count

Common pitfalls

• Using unfiltered/noisy versions of the datasets without applying Shi et al.'s cleaning templates, which introduces erroneous documentation and degrades baseline performance.
• Comparing zero-shot (no fine-tuning) vs fine-tuned configurations without explicit labeling, as zero-shot settings perform significantly worse and mask true model capability.
• Ignoring PCA dimension reduction effects (dim=20 vs dim=768) which drastically change baseline MRR scores and must be reported separately for fair comparison.

Evidence (verbatim from paper)

Mean Reciprocal Rank (MRR) is used for code search evaluation in many approaches. The effect in MRR over code search on set C of cases by ASTTrans embedding a to original embedding model o with original embedding size d, is calculated by Formula [6]: EffectMRR(C,a,o,d)=MRR_com(C,a,o,d)−MRR_org(C,o,d). In this Formula, MRR_org is the Original MRR returned by the original code search. MRR_com is the Combined MRR returned by code search with the combined similarity matrix for a specific set of queries C, the original model o reduced to dimension size d by PCA and the augmented model a. If the score of this metric is positive, it means the augmented code search process improves the accuracy of the original code search process by MRR.

Citation

@misc{phan2023evaluating,
  title={Evaluating and Optimizing the Effectiveness of Neural Machine Translation in Supporting Code Retrieval Models: A Study on the CAT Benchmark},
  author={Hung Phan et al. (2023)},
  year={2023},
  note={arXiv:2308.04693}
}

• arXiv: 2308.04693