Codenet Classification Eval

codenet-classification-eval

CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks — Puri et al. (2021) (arXiv:2105.12655, 2021)

What this evaluates

Evaluates a model's ability to classify source code into the programming problem it was submitted to solve. It probes code representation learning and structural understanding by mapping code samples to their corresponding problem classes.

Datasets

• CodeNet (Java250, Python800, C++1000, C++1400) — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/IBM/Project_CodeNet

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified code samples out of the total test samples. Calculated as (correct predictions / total test instances) * 100.

Input / output format

Input: Source code snippet (represented as token sequence, bag-of-tokens, or graph structure) corresponding to a programming problem submission.

Output: Predicted problem class label (integer index or string name of the coding problem).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• C++ benchmarks contain noise from copied constants and data structures across different problem submissions, which can artificially lower model performance.
• Tokenization strategy heavily impacts results; using a language-specific tokenizer (e.g., Clang for C) vs. generic tokenizers changes the feature space significantly.
• The 80/20 split is random, which may cause data leakage if similar code submissions are split across train and test sets.

Evidence (verbatim from paper)

In each experiment, 20% of the code samples are used for testing, while the rest are split in 4:1 for training and validation, respectively. Table 7 summarizes the classification accuracy for all models on all benchmarks.

Citation

@misc{puri2021codenet,
  title={CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks},
  author={Puri et al. (2021)},
  year={2021},
  note={arXiv:2105.12655}
}

• arXiv: 2105.12655