qhjqhj00 clone-detection-eval
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation — Lu et al. (2021) (arXiv:2102.04664, 2021)
What this evaluates
Evaluates a model's ability to determine whether two code snippets share the same semantics or to retrieve relevant code snippets from a repository. It probes semantic code similarity and code retrieval capabilities.
Datasets
- BigCloneBench — total ?; splits: test (-1)
- POJ-104 — total ?; splits: test (-1)
Metrics
F1— range: [0, 1]- Harmonic mean of precision and recall for binary classification of code pair similarity.
MAP— range: [0, 1]- Mean Average Precision for retrieving 499 code snippets.
Overall(primary) — range: [0, 1]- Average of F1 (BigCloneBench) and MAP (POJ-104) scores.
Input / output format
Input: Pairs of code snippets (BigCloneBench) or a query code snippet with a candidate pool (POJ-104).
Output: Binary label (same/different semantics) or ranked list of retrieved code snippets.
Scoring recipe
For BigCloneBench: compute F1 on binary predictions.
For POJ-104: compute MAP over retrieved candidates.
Overall = (F1 + MAP) / 2
Common pitfalls
- Models often ignore code structure (ASTs, data flow) which significantly impacts similarity measurement.
- POJ-104 retrieval task requires ranking 499 candidates, not just binary classification.
Evidence (verbatim from paper)
The task of the BigCloneBench dataset is formulated as a binary classification to predict whether a given pair of codes has the same semantics, with the F1 score used as the evaluation metric. The task of the POJ-104 dataset aims to retrieve 499 codes for a given code from the development/test set for validation/testing, with the Mean Average Precision (MAP) as the evaluation metric. The overall score of the clone detection task is the average value of F1 and MAP scores.
Citation
@misc{lu2021codexglue,
title={CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation},
author={Lu et al. (2021)},
year={2021},
note={arXiv:2102.04664}
}
- arXiv: 2102.04664