Code Pretraining Impact Eval

code-pretraining-impact-eval

To Code, or Not To Code? Exploring Impact of Code in Pre-training — Arya et al. (2024) (arXiv:2408.10914, 2024)

What this evaluates

This evaluation protocol measures the impact of code data proportions and quality during LLM pre-training on downstream capabilities. It probes natural language reasoning, world knowledge, code generation, and generative text quality across different model initialization and pre-training mixture variants.

Datasets

• NL Reasoning Benchmarks — total ?; splits: test (-1)
• World Knowledge Tasks — total ?; splits: test (-1)
• Code Benchmarks (Python) — total ?; splits: test (-1)
• Dolly-200-English — total ?; splits: test (-1)

Metrics

• pass@1 (primary) — range: percent
  • The fraction of generated code solutions that pass all provided unit tests on the first attempt. Reported as a percentage.
• win-rate — range: percent
  • Pairwise comparison of model generations against a baseline on open-ended prompts. Calculated as the percentage of wins minus losses relative to total comparisons.
• relative improvement — range: percent
  • Percentage change in metric score compared to a baseline model: ((new_score - baseline_score) / baseline_score) * 100.

Input / output format

Input: Natural language questions, reasoning prompts, or coding tasks presented as text prompts.

Output: Generated text responses or Python code solutions.

Scoring recipe

def score_code(predictions, gold_tests):
    passed = 0
    for pred in predictions:
        if run_tests(pred, gold_tests): passed += 1
    return (passed / len(predictions)) * 100

def score_winrate(model_a, model_b, prompts):
    wins, losses, ties = 0, 0, 0
    for p in prompts:
        res = judge(model_a.generate(p), model_b.generate(p))
        if res == 'A': wins += 1
        elif res == 'B': losses += 1
        else: ties += 1
    return ((wins - losses) / (wins + losses + ties)) * 100

Common pitfalls

• The paper reports relative improvements rather than absolute scores in the text, which can obscure baseline performance levels.
• Code evaluation is specifically measured in Python, so results may not generalize to other programming languages without re-evaluation.
• Win-rates are calculated against a text-only baseline, not an absolute standard, making cross-paper comparisons difficult.

Evidence (verbatim from paper)

In code evaluation, there is a linear increase in performance as the amount of code increases, with the best model being a code-only model. As observable in Figure [4] right inset, the 100% code leads to a 2.6x increase in the code benchmarks compared to the 25% code model. As expected, for the model with 0% code, the average pass@1 score drops to 0.

Citation

@misc{arya2024tocodeornotcode,
  title={To Code, or Not To Code? Exploring Impact of Code in Pre-training},
  author={Arya et al. (2024)},
  year={2024},
  note={arXiv:2408.10914}
}

• arXiv: 2408.10914