Concode Eval

concode-eval

Mapping Language to Code in Programmatic Context — Iyer et al. (2018) (arXiv:1808.09588, 2018)

What this evaluates

Probes a model's ability to generate syntactically valid Java member functions from natural language documentation, conditioned on a full class environment including variable types, method signatures, and their interdependencies. It evaluates context-aware code generation, identifier disambiguation, and code reusability.

Datasets

• CONCODE — total 100000; splits: train (-1), val (-1), test (2000); repo https://github.com/sriniyer/concode

Metrics

• Exact match accuracy (primary) — range: [0, 1]
  • 1 if the generated code exactly matches the reference code, else 0. Averaged over the dataset.
• BLEU — range: [0, 1] or percent
  • An n-gram precision-based metric that measures the overlap between predicted and reference code tokens to provide partial credit for syntactically similar outputs.

Input / output format

Input: Natural language documentation combined with programmatic context (variable types, method signatures, and their interdependencies), restricted to ≤200 tokens total.

Output: Java member function source code (generated via syntactically valid production rules), restricted to ≤150 tokens or ≤500 production rules.

Scoring recipe

def compute_metrics(predictions, references):
    exact_matches = sum(1 for p, r in zip(predictions, references) if p == r)
    exact_acc = exact_matches / len(references)
    bleu = compute_bleu(references, predictions)  # standard n-gram precision
    return exact_acc, bleu

Common pitfalls

• Exact match is extremely strict and penalizes minor formatting or whitespace differences, often underestimating functional correctness.
• BLEU score focuses on n-gram overlap and may assign high scores to syntactically valid but semantically incorrect code.
• Evaluation requires strict tokenization (camel-case splitting, lower-casing) which significantly impacts metric scores if not replicated.

Evidence (verbatim from paper)

To evaluate the quality of the output, we use Exact match accuracy between the reference and generated code. As a measure of partial credit, we also compute the BLEU score (Papineni et al., 2002), following recent work on code generation (Ling et al., 2016; Yin and Neubig, 2017). BLEU is an n-gram precision-based metric that will be higher when more subparts of the predicted code match the provided reference.

Citation

@misc{iyer2018mapping,
  title={Mapping Language to Code in Programmatic Context},
  author={Iyer et al. (2018)},
  year={2018},
  note={arXiv:1808.09588}
}

• arXiv: 1808.09588