Phi 1 Textbooks Are All You Need Arxiv 2306 11644v2

Phi-1: Textbooks Are All You Need

One-line decision

Use this skill when you want to generate synthetic textbook-quality code data for training a small but highly capable code model. Avoid it when you have sufficient high-quality code data or are not training a code model.

Skill metadata

• Skill type: code-textbook-synthesis
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Generate synthetic textbook-quality code data using GPT-3.5, demonstrating that 1.3B models trained on high-quality synthetic data can match much larger models on code benchmarks.

Problem signature

• Modality: synthetic code textbook data generated by LLMs.
• Data state: synthetic code exercises and textbook content generated by GPT-3.5.
• Scale regime: 6B tokens of synthetic + filtered code data.
• Model requirement: 1.3B parameter code model.

Use when

• You want high-quality code training data.
• You can generate synthetic textbook content.
• You are training a small code model.

Do not use when

• You have sufficient code data from GitHub.
• You are not training a code model.
• Synthetic code quality is insufficient.

Required inputs

• synthetic_generator: GPT-3.5 for generating code textbook content.
• quality_filters: Filters for selecting textbook-quality code.
• base_model: 1.3B parameter model for training.

Optional inputs

• filtered_web_code: Filtered web code for supplementation.

Outputs

• code_textbooks: Synthetic textbook-quality code data.
• phi_1: 1.3B code model with strong performance.

Assumptions and prerequisites

• Synthetic textbook code is higher quality than raw GitHub code.
• Small models benefit most from high-quality data.
• GPT-3.5 can generate useful code teaching examples.

Procedure

1. Generate code textbooks Action: Use GPT-3.5 to generate textbook-quality code examples. Why: Creates high-quality code training data. Note: See paper for details.
2. Filter web code Action: Select textbook-quality code from web sources. Why: Supplements synthetic data with real code. Note: See paper for details.
3. Train Phi-1 Action: Train a 1.3B model on synthetic + filtered code. Why: Validates quality-over-quantity approach. Note: See paper for details.

Parameters to set

• synthetic_tokens — Role: Tokens of synthetic code data. How to set: 1B+ synthetic tokens. Default/range: 1B. Effect: More synthetic data improves code quality.
• total_tokens — Role: Total training tokens. How to set: 6B combined. Default/range: 6B. Effect: Quality matters more than quantity.

Validation checks

• Phi-1 should outperform larger models on HumanEval.
• Synthetic code should be correct and educational.
• Quality data should compensate for small model size.

Failure modes

• Synthetic code may have errors.
• The approach may not generalize beyond code.
• GPT-3.5 generation has cost.

Adaptation notes for VLM training

• Synthetic textbook generation applies to visual reasoning training data.
• The quality-over-quantity insight transfers to VLM data.
• Generate educational content for domain-specific VLM training.

Implementation notes

• Generate diverse code examples.
• Filter rigorously for quality.
• Benchmark against larger models.

Evidence from the paper

• Phi-1 (1.3B) achieves 50.6% on HumanEval through synthetic textbook data.
• Textbook-quality data enables small models to match larger ones.
• The approach demonstrates that data quality can substitute for model scale.
• Phi-1 pioneered the synthetic textbook data approach.

Source paper

• Title: Phi-1: Textbooks Are All You Need
• Year: 2023
• Venue: arXiv
• Paper ID: arxiv-2306.11644v2
• URL: http://arxiv.org/abs/2306.11644v2
• arXiv ID: 2306.11644v2