Phi 15 Textbooks Are All You Need Ii Arxiv 2309 05463v2

Phi-1.5: Textbooks Are All You Need II

One-line decision

Use this skill when you want to generate synthetic textbook-quality training data to teach reasoning to small language models. Avoid it when you are training a large model where data quality filtering alone suffices.

Skill metadata

• Skill type: synthetic-textbook-for-reasoning
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Generate synthetic textbook-quality training data using a strong LLM to teach common-sense reasoning to a 1.3B parameter model, demonstrating that data quality can substitute for scale.

Problem signature

• Modality: synthetic textbook-quality text for reasoning training.
• Data state: LLM-generated educational content for reasoning training.
• Scale regime: 30B tokens of synthetic and filtered data.
• Model requirement: Phi-1.5 1.3B parameter model.

Use when

• You want to teach reasoning to a small model.
• You can generate synthetic educational content.
• Data quality matters more than quantity.

Do not use when

• You are training a very large model.
• You have sufficient high-quality reasoning data.
• Synthetic data generation is too expensive.

Required inputs

• synthetic_generator: Strong LLM for generating textbook content.
• quality_filters: Filters for selecting educational-quality content.
• base_model: Small model (1.3B) for training.

Optional inputs

• web_filtered: Heavily filtered web data supplement.

Outputs

• synthetic_textbooks: Synthetic textbook-quality training data.
• phi_1_5: 1.3B model with strong reasoning.

Assumptions and prerequisites

• Textbook-quality data teaches reasoning more effectively.
• Synthetic generation can produce educational content.
• Small models benefit most from high-quality data.

Procedure

1. Generate synthetic textbooks Action: Use a strong LLM to generate educational text. Why: Creates textbook-quality training data. Note: See paper for details.
2. Filter web data Action: Select educational content from web data. Why: Supplements synthetic data with real content. Note: See paper for details.
3. Combine data sources Action: Mix synthetic and filtered data. Why: Both contribute complementary quality. Note: See paper for details.
4. Train Phi-1.5 Action: Train the 1.3B model on curated data. Why: Validates quality-over-quantity approach. Note: See paper for details.

Parameters to set

• synthetic_fraction — Role: Fraction of synthetic data. How to set: Significant fraction. Default/range: Large. Effect: More synthetic data improves reasoning.
• total_tokens — Role: Total training tokens. How to set: 30B. Default/range: 30B. Effect: Quality matters more than token count at small scale.

Validation checks

• Phi-1.5 should outperform larger models on reasoning.
• Synthetic data should improve reasoning benchmarks.
• The approach should be cost-effective.

Failure modes

• Synthetic data may lack diversity.
• Small models have inherent limitations.
• Quality filtering is expensive.

Adaptation notes for VLM training

• Apply synthetic textbook generation to VLM visual reasoning training.
• Use quality-focused data curation for small VLMs.
• The approach validates data quality over quantity.

Implementation notes

• Generate diverse educational content.
• Validate reasoning quality on benchmarks.
• Compare to web-only baselines.

Evidence from the paper

• Phi-1.5 (1.3B) achieves reasoning comparable to much larger models through data quality.
• Synthetic textbook data is more effective than raw web data for reasoning.
• Data quality can substitute for model scale.
• 30B high-quality tokens outperform much larger low-quality corpora.

Source paper

• Title: Phi-1.5: Textbooks Are All You Need II
• Year: 2023
• Venue: arXiv
• Paper ID: arxiv-2309.05463v2
• URL: http://arxiv.org/abs/2309.05463v2
• arXiv ID: 2309.05463v2