Nl Code Pair Mining Eval

nl-code-pair-mining-eval

Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow — Pengcheng Yin et al. (arXiv:1805.08949, 2018)

What this evaluates

Evaluates a machine learning model's ability to automatically extract high-quality, aligned natural language intent and code snippet pairs from Stack Overflow posts. It probes the system's ranking capability, precision, and recall across different programming languages and feature combinations.

Datasets

• Stack Overflow NL-Code Pairs — total ?; splits: train (71828), val (7981), annotated (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Curve (AUC) computed over Precision-Recall and ROC curves. PR plots precision vs recall for top-k predictions; ROC plots true positive rate vs false positive rate.
• Precision-Recall (PR) — range: [0, 1]
  • Curve plotting precision against recall for top-k ranked predictions, where k ranges from 1 to the total number of candidates.
• ROC — range: [0, 1]
  • Receiver Operating Characteristic curve plotting true positive rate against false positive rate for the ranked candidate list.

Input / output format

Input: Stack Overflow question title (intent) and its answers containing code blocks.

Output: Ranked list of candidate intent/snippet pairs with probability scores.

Scoring recipe

# For each SO question:
# 1. Extract candidate intent/snippet pairs from answers
# 2. Score and rank candidates by model probability
# 3. Compare ranked list against gold-standard annotations
# 4. Compute precision and recall at each rank k
# 5. Plot PR and ROC curves across all questions
# 6. Calculate AUC for both curves as summary metric

Common pitfalls

• Correspondence features trained on full code blocks bias towards imports and assignments, hurting fine-grained snippet extraction.
• Structural features over-prioritize full code blocks or last lines, missing partial implementations.
• Cross-language transfer requires feature normalization due to different value ranges across languages.
• Gold annotations may be incomplete, leading to false negatives in evaluation.

Evidence (verbatim from paper)

We present the results using standard precision-recall (PR) and Receiver Operating Characteristic (ROC) curves. In short, a PR curve shows the precision w.r.t. recall for the top- k predictions in the ranked list, with k from 1 to the number of candidates. A ROC curve plots the true positive rates w.r.t. false positive rates in similar fashion. We also compute the Area Under the Curve (AUC) scores for all ROC curves.

Citation

@misc{yin2018learning,
  title={Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow},
  author={Pengcheng Yin et al.},
  year={2018},
  note={arXiv:1805.08949}
}

• arXiv: 1805.08949