Deita What Makes Good Data For Alignment Arxiv 2312 15685v2

Deita: What Makes Good Data for Alignment

One-line decision

Use this skill when you want to select the best alignment data from a large pool using complexity and quality scoring. Avoid it when you do not have a large pool of alignment data to select from.

Skill metadata

• Skill type: alignment-data-selection
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Select the most effective alignment data from a large pool by scoring each example on complexity and quality, then selecting a diverse, high-scoring subset for efficient alignment.

Problem signature

• Modality: text instruction-response pairs scored for quality and complexity.
• Data state: large pool of alignment data scored and filtered for selection.
• Scale regime: 6K-10K selected from 300K+ candidates.
• Model requirement: Any LLM for alignment; scorers for quality and complexity.

Use when

• You have a large pool of alignment data.
• You want to select the most effective subset.
• You can score data on quality and complexity.

Do not use when

• Your alignment data pool is small.
• Random selection is sufficient.
• You cannot compute quality/complexity scores.

Required inputs

• alignment_pool: Large pool of instruction-response pairs.
• quality_scorer: Model for scoring response quality.
• complexity_scorer: Model for scoring instruction complexity.

Optional inputs

• diversity_filter: Filter for ensuring selected data diversity.

Outputs

• selected_data: 6K-10K high-quality, complex alignment samples.
• aligned_model: Model aligned on the selected data.

Assumptions and prerequisites

• Quality and complexity are key dimensions for alignment data value.
• A small, well-selected subset can match larger datasets.
• Scoring enables principled data selection.

Procedure

1. Score quality and complexity Action: Score each example for response quality and instruction complexity. Why: Quality and complexity predict alignment value. Note: See paper for details.
2. Select top examples Action: Select examples with highest combined quality-complexity scores. Why: High-scoring examples provide the best alignment signal. Note: See paper for details.
3. Ensure diversity Action: Apply diversity filtering to the selected set. Why: Diversity prevents overfitting to specific patterns. Note: See paper for details.
4. Align model Action: Fine-tune the model on the selected 6K-10K examples. Why: Validates that selection improves alignment efficiency. Note: See paper for details.

Parameters to set

• selection_size — Role: Number of examples to select. How to set: 6K-10K for efficient alignment. Default/range: 6K-10K. Effect: Smaller, better-selected data can match larger datasets.
• quality_weight — Role: Weight of quality in scoring. How to set: Balance with complexity. Default/range: Balanced. Effect: Affects which examples are selected.
• complexity_weight — Role: Weight of complexity in scoring. How to set: Balance with quality. Default/range: Balanced. Effect: Higher weight selects harder examples.

Validation checks

• Selected data should outperform random selection.
• 6K-10K selected examples should match larger dataset performance.
• Quality and complexity scores should correlate with alignment value.

Failure modes

• Scoring models may not perfectly predict alignment value.
• Small selected sets may miss important instruction types.
• Over-emphasis on complexity may select confusing examples.

Adaptation notes for VLM training

• Apply Deita scoring to VLM instruction tuning data selection.
• Use complexity and quality scoring for any alignment data.
• The scoring approach generalizes to multimodal data.

Implementation notes

• Train quality and complexity scorers on rated data.
• Use the Deita pipeline for reproducibility.
• Compare selected data to full data and random baselines.

Evidence from the paper

• Deita selects 6K-10K examples that match 300K+ dataset performance.
• Quality and complexity scoring effectively identifies valuable alignment data.
• Data selection for alignment significantly improves efficiency.
• Small, well-selected datasets can match or exceed much larger ones.

Source paper

• Title: Deita: What Makes Good Data for Alignment
• Year: 2024
• Venue: ICLR
• Paper ID: arxiv-2312.15685v2
• URL: http://arxiv.org/abs/2312.15685v2
• arXiv ID: 2312.15685v2