Minicpm V A Gpt 4v Level Mllm On Your Phone Arxiv 2408 01800v3

MiniCPM-V: A GPT-4V Level MLLM on Your Phone

One-line decision

Use this skill when you want a data recipe for training a phone-deployable VLM that approaches GPT-4V quality on key benchmarks. Avoid it when you are not targeting mobile deployment.

Skill metadata

• Skill type: mobile-vlm-data-recipe
• Paper kind: operational-method
• Actionability: high
• Evidence quality: full_paper

Goal

Design a data recipe for training a compact VLM deployable on phones that approaches GPT-4V quality, emphasizing data quality and efficient architecture.

Problem signature

• Modality: high-quality instruction data for compact VLM training.
• Data state: curated high-quality instruction data for mobile VLM.
• Scale regime: curated instruction data emphasizing quality.
• Model requirement: MiniCPM (2.4B) + SigLIP ViT.

Use when

• You want phone-deployable VLM quality.
• Data quality is critical for your compact model.
• You need efficient VLM deployment.

Do not use when

• Maximum capability without size constraint.
• Mobile deployment is not your target.
• You have unlimited compute.

Required inputs

• high_quality_data: Curated instruction data for compact training.
• compact_llm: MiniCPM 2.4B language model.
• efficient_encoder: SigLIP ViT for visual encoding.

Optional inputs

• quantization: Quantization for mobile deployment.

Outputs

• minicpmv_model: Phone-deployable VLM.
• mobile_data_recipe: Data recipe for compact VLMs.

Assumptions and prerequisites

• High-quality data enables compact models to approach larger model quality.
• Mobile deployment requires extreme efficiency.
• Quality-focused data curation is most important for small models.

Procedure

1. Curate high-quality data Action: Select the highest quality instruction data available. Why: Quality is critical for compact models. Note: See paper for details.
2. Train compact VLM Action: Train MiniCPM-V on curated data. Why: Validates the compact data recipe. Note: See paper for details.
3. Optimize for mobile Action: Apply quantization and optimization for phone deployment. Why: Enables actual mobile deployment. Note: See paper for details.
4. Evaluate quality Action: Compare to GPT-4V on key benchmarks. Why: Validates approaching frontier quality. Note: See paper for details.

Parameters to set

• model_size — Role: Compact model size. How to set: 2.4B for mobile. Default/range: 2.4B. Effect: Small enough for phones.
• data_quality — Role: Quality of training data. How to set: Maximally curated. Default/range: Highest quality. Effect: Quality compensates for small model size.

Validation checks

• Should approach GPT-4V on key benchmarks.
• Should deploy on phones.
• Data quality should demonstrably drive performance.

Failure modes

• 2.4B may be insufficient for complex tasks.
• Mobile optimization may reduce quality.
• Quality data curation is expensive.

Adaptation notes for VLM training

• MiniCPM-V demonstrates compact VLM viability.
• The data recipe is transferable to other compact architectures.
• Quality-focused curation is the key insight.

Implementation notes

• Use INT4 quantization for mobile.
• Focus on data quality metrics.
• Benchmark against GPT-4V.

Evidence from the paper

• MiniCPM-V approaches GPT-4V quality at 2.4B parameters.
• Data quality is the primary driver of compact model capability.
• The model deploys on phones with quantization.
• Compact VLMs with quality data are viable for mobile.

Source paper

• Title: MiniCPM-V: A GPT-4V Level MLLM on Your Phone
• Year: 2024
• Venue: arXiv
• Paper ID: arxiv-2408.01800v3
• URL: http://arxiv.org/abs/2408.01800v3
• arXiv ID: 2408.01800v3