Codebleu

codebleu

CodeBLEU: a Method for Automatic Evaluation of Code Synthesis — Ren et al. (2020) (arXiv:2009.10297, 2020)

What this evaluates

Evaluates the validity of the CodeBLEU metric for code synthesis by measuring its correlation with human programmer judgments across text-to-code generation, code translation, and code refinement tasks.

Datasets

• Iyer et al. [2018] dataset — total 104000; splits: train (100000), val (2000), test (2000)
• Code translation dataset (Java to C#) — total 11800; splits: dev (500), test (1000)
• Tufano et al. [2019] dataset — total ?; splits: train_small (46680), dev_small (5835), test_small (5835), train_medium (52364), dev_medium (6545), test_medium (6545)

Metrics

• CodeBLEU (primary) — range: [0, 100]
  • Combines weighted n-gram matching (BLEU), AST subtree alignment, and semantic similarity via data-flow analysis using hyperparameters α, β, γ, δ. Scaled to 0-100.

Input / output format

Input: Natural language documentation and class environment (variable names/types, method names/return types) for text-to-code; Java method code for translation; buggy Java function code for refinement.

Output: Generated code (Java method/function or C# method).

Scoring recipe

def compute_codebleu(predictions, references):
    bleu = compute_ngram_precision(predictions, references)
    ast_match = compute_ast_subtree_match(predictions, references)
    df_match = compute_dataflow_similarity(predictions, references)
    # Default weights: alpha=0.25, beta=0.25, gamma=0.25, delta=0.25
    score = 0.25 * bleu + 0.25 * ast_match + 0.25 * df_match
    return score * 100  # Paper reports scores in [0, 100]

Common pitfalls

• CodeBLEU scores are scaled to 0-100, not 0-1.
• Human evaluation uses a 1-5 Likert scale, not a percentage or 0-1 score.
• The metric's validity is assessed via Pearson correlation with human scores, not direct task accuracy.
• Hyperparameters α, β, γ, δ significantly impact correlation; default is not always optimal.

Evidence (verbatim from paper)

For each task, we calculate the Pearson correlation coefficient to check the correlation between the scores given by our proposed CodeBLEU and the scores assigned by programmers (human evaluation scores).

Citation

@misc{ren2020codebleu,
  title={CodeBLEU: a Method for Automatic Evaluation of Code Synthesis},
  author={Ren et al. (2020)},
  year={2020},
  note={arXiv:2009.10297}
}

• arXiv: 2009.10297