AI Multimodal Processing Skill
Process audio, images, videos, documents, and generate images using Google Gemini's multimodal API. Unified interface for all multimedia content understanding and generation.
Core Capabilities
Audio Processing
- Transcription with timestamps (up to 9.5 hours)
- Audio summarization and analysis
- Speech understanding and speaker identification
- Music and environmental sound analysis
- Text-to-speech generation with controllable voice
Image Understanding
- Image captioning and description
- Object detection with bounding boxes (2.0+)
- Pixel-level segmentation (2.5+)
- Visual question answering
- Multi-image comparison (up to 3,600 images)
- OCR and text extraction
Video Analysis
- Scene detection and summarization
- Video Q&A with temporal understanding
- Transcription with visual descriptions
- YouTube URL support
- Long video processing (up to 6 hours)
- Frame-level analysis
Document Extraction
- Native PDF vision processing (up to 1,000 pages)
- Table and form extraction
- Chart and diagram analysis
- Multi-page document understanding
- Structured data output (JSON schema)
- Format conversion (PDF to HTML/JSON)
Image Generation
- Text-to-image generation
- Image editing and modification
- Multi-image composition (up to 3 images)
- Iterative refinement
- Multiple aspect ratios (1:1, 16:9, 9:16, 4:3, 3:4)
- Controllable style and quality
Capability Matrix
| Task |
Audio |
Image |
Video |
Document |
Generation |
| Transcription |
✓ |
- |
✓ |
- |
- |
| Summarization |
✓ |
✓ |
✓ |
✓ |
- |
| Q&A |
✓ |
✓ |
✓ |
✓ |
- |
| Object Detection |
- |
✓ |
✓ |
- |
- |
| Text Extraction |
- |
✓ |
- |
✓ |
- |
| Structured Output |
✓ |
✓ |
✓ |
✓ |
- |
| Creation |
TTS |
- |
- |
- |
✓ |
| Timestamps |
✓ |
- |
✓ |
- |
- |
| Segmentation |
- |
✓ |
- |
- |
- |
Model Selection Guide
Gemini 2.5 Series (Recommended)
- gemini-2.5-pro: Highest quality, all features, 1M-2M context
- gemini-2.5-flash: Best balance, all features, 1M-2M context
- gemini-2.5-flash-lite: Lightweight, segmentation support
- gemini-2.5-flash-image: Image generation only
Feature Requirements
- Segmentation: Requires 2.5+ models
- Object Detection: Requires 2.0+ models
- Multi-video: Requires 2.5+ models
- Image Generation: Requires flash-image model
Context Windows
- 2M tokens: ~6 hours video (low-res) or ~2 hours (default)
- 1M tokens: ~3 hours video (low-res) or ~1 hour (default)
- Audio: 32 tokens/second (1 min = 1,920 tokens)
- PDF: 258 tokens/page (fixed)
- Image: 258-1,548 tokens based on size
Quick Start
Prerequisites
API Key Setup: Supports both Google AI Studio and Vertex AI.
The skill checks for GEMINI_API_KEY in this order:
- Process environment:
export GEMINI_API_KEY="your-key"
- Project root:
.env
.claude/.env.claude/skills/.env.claude/skills/ai-multimodal/.env
Get API key: https://aistudio.google.com/apikey
For Vertex AI:
export GEMINI_USE_VERTEX=true
export VERTEX_PROJECT_ID=your-gcp-project-id
export VERTEX_LOCATION=us-central1 # Optional
Install SDK:
pip install google-genai python-dotenv pillow
Common Patterns
Transcribe Audio:
python scripts/gemini_batch_process.py \
--files audio.mp3 \
--task transcribe \
--model gemini-2.5-flash
Analyze Image:
python scripts/gemini_batch_process.py \
--files image.jpg \
--task analyze \
--prompt "Describe this image" \
--output docs/assets/<output-name>.md \
--model gemini-2.5-flash
Process Video:
python scripts/gemini_batch_process.py \
--files video.mp4 \
--task analyze \
--prompt "Summarize key points with timestamps" \
--output docs/assets/<output-name>.md \
--model gemini-2.5-flash
Extract from PDF:
python scripts/gemini_batch_process.py \
--files document.pdf \
--task extract \
--prompt "Extract table data as JSON" \
--output docs/assets/<output-name>.md \
--format json
Generate Image:
python scripts/gemini_batch_process.py \
--task generate \
--prompt "A futuristic city at sunset" \
--output docs/assets/<output-file-name> \
--model gemini-2.5-flash-image \
--aspect-ratio 16:9
Optimize Media:
# Prepare large video for processing
python scripts/media_optimizer.py \
--input large-video.mp4 \
--output docs/assets/<output-file-name> \
--target-size 100MB
# Batch optimize multiple files
python scripts/media_optimizer.py \
--input-dir ./videos \
--output-dir docs/assets/optimized \
--quality 85
Convert Documents to Markdown:
# Convert to PDF
python scripts/document_converter.py \
--input document.docx \
--output docs/assets/document.md
# Extract pages
python scripts/document_converter.py \
--input large.pdf \
--output docs/assets/chapter1.md \
--pages 1-20
Supported Formats
Audio
- WAV, MP3, AAC, FLAC, OGG Vorbis, AIFF
- Max 9.5 hours per request
- Auto-downsampled to 16 Kbps mono
Images
- PNG, JPEG, WEBP, HEIC, HEIF
- Max 3,600 images per request
- Resolution: ≤384px = 258 tokens, larger = tiled
Video
- MP4, MPEG, MOV, AVI, FLV, MPG, WebM, WMV, 3GPP
- Max 6 hours (low-res) or 2 hours (default)
- YouTube URLs supported (public only)
Documents
- PDF only for vision processing
- Max 1,000 pages
- TXT, HTML, Markdown supported (text-only)
Size Limits
- Inline: <20MB total request
- File API: 2GB per file, 20GB project quota
- Retention: 48 hours auto-delete
Reference Navigation
For detailed implementation guidance, see:
Audio Processing
references/audio-processing.md - Transcription, analysis, TTS
- Timestamp handling and segment analysis
- Multi-speaker identification
- Non-speech audio analysis
- Text-to-speech generation
Image Understanding
references/vision-understanding.md - Captioning, detection, OCR
- Object detection and localization
- Pixel-level segmentation
- Visual question answering
- Multi-image comparison
Video Analysis
references/video-analysis.md - Scene detection, temporal understanding
- YouTube URL processing
- Timestamp-based queries
- Video clipping and FPS control
- Long video optimization
Document Extraction
references/document-extraction.md - PDF processing, structured output
- Table and form extraction
- Chart and diagram analysis
- JSON schema validation
- Multi-page handling
Image Generation
references/image-generation.md - Text-to-image, editing
- Prompt engineering strategies
- Image editing and composition
- Aspect ratio selection
- Safety settings
Cost Optimization
Token Costs
Input Pricing:
- Gemini 2.5 Flash: $1.00/1M input, $0.10/1M output
- Gemini 2.5 Pro: $3.00/1M input, $12.00/1M output
- Gemini 1.5 Flash: $0.70/1M input, $0.175/1M output
Token Rates:
- Audio: 32 tokens/second (1 min = 1,920 tokens)
- Video: ~300 tokens/second (default) or ~100 (low-res)
- PDF: 258 tokens/page (fixed)
- Image: 258-1,548 tokens based on size
TTS Pricing:
- Flash TTS: $10/1M tokens
- Pro TTS: $20/1M tokens
Best Practices
- Use
gemini-2.5-flash for most tasks (best price/performance)
- Use File API for files >20MB or repeated queries
- Optimize media before upload (see
media_optimizer.py)
- Process specific segments instead of full videos
- Use lower FPS for static content
- Implement context caching for repeated queries
- Batch process multiple files in parallel
Rate Limits
Free Tier:
- 10-15 RPM (requests per minute)
- 1M-4M TPM (tokens per minute)
- 1,500 RPD (requests per day)
YouTube Limits:
- Free tier: 8 hours/day
- Paid tier: No length limits
- Public videos only
Storage Limits:
- 20GB per project
- 2GB per file
- 48-hour retention
Error Handling
Common errors and solutions:
- 400: Invalid format/size - validate before upload
- 401: Invalid API key - check configuration
- 403: Permission denied - verify API key restrictions
- 404: File not found - ensure file uploaded and active
- 429: Rate limit exceeded - implement exponential backoff
- 500: Server error - retry with backoff
Scripts Overview
All scripts support unified API key detection and error handling:
gemini_batch_process.py: Batch process multiple media files
- Supports all modalities (audio, image, video, PDF)
- Progress tracking and error recovery
- Output formats: JSON, Markdown, CSV
- Rate limiting and retry logic
- Dry-run mode
media_optimizer.py: Prepare media for Gemini API
- Compress videos/audio for size limits
- Resize images appropriately
- Split long videos into chunks
- Format conversion
- Quality vs size optimization
document_converter.py: Convert documents to PDF
- Convert DOCX, XLSX, PPTX to PDF
- Extract page ranges
- Optimize PDFs for Gemini
- Extract images from PDFs
- Batch conversion support
Run any script with --help for detailed usage.
Resources
1---2name: ai-multimodal3description: Process and generate multimedia content using Google Gemini API. Capabilities include analyze audio files (transcription with timestamps, summarization, speech understanding, music/sound analysis up to 9.5 hours), understand images (captioning, object detection, OCR, visual Q&A, segmentation), process videos (scene detection, Q&A, temporal analysis, YouTube URLs, up to 6 hours), extract from documents (PDF tables, forms, charts, diagrams, multi-page), generate images (text-to-image, editing, composition, refinement). Use when working with audio/video files, analyzing images or screenshots, processing PDF documents, extracting structured data from media, creating images from text prompts, or implementing multimodal AI features. Supports multiple models (Gemini 2.5/2.0) with context windows up to 2M tokens.4license: MIT5---6
7# AI Multimodal Processing Skill
8
9Process audio, images, videos, documents, and generate images using Google Gemini's multimodal API. Unified interface for all multimedia content understanding and generation.
10
11## Core Capabilities
12
13### Audio Processing
14- Transcription with timestamps (up to 9.5 hours)
15- Audio summarization and analysis
16- Speech understanding and speaker identification
17- Music and environmental sound analysis
18- Text-to-speech generation with controllable voice
19
20### Image Understanding
21- Image captioning and description
22- Object detection with bounding boxes (2.0+)
23- Pixel-level segmentation (2.5+)
24- Visual question answering
25- Multi-image comparison (up to 3,600 images)
26- OCR and text extraction
27
28### Video Analysis
29- Scene detection and summarization
30- Video Q&A with temporal understanding
31- Transcription with visual descriptions
32- YouTube URL support
33- Long video processing (up to 6 hours)
34- Frame-level analysis
35
36### Document Extraction
37- Native PDF vision processing (up to 1,000 pages)
38- Table and form extraction
39- Chart and diagram analysis
40- Multi-page document understanding
41- Structured data output (JSON schema)
42- Format conversion (PDF to HTML/JSON)
43
44### Image Generation
45- Text-to-image generation
46- Image editing and modification
47- Multi-image composition (up to 3 images)
48- Iterative refinement
49- Multiple aspect ratios (1:1, 16:9, 9:16, 4:3, 3:4)
50- Controllable style and quality
51
52## Capability Matrix
53
54| Task | Audio | Image | Video | Document | Generation |
55|------|:-----:|:-----:|:-----:|:--------:|:----------:|
56| Transcription | ✓ | - | ✓ | - | - |
57| Summarization | ✓ | ✓ | ✓ | ✓ | - |
58| Q&A | ✓ | ✓ | ✓ | ✓ | - |
59| Object Detection | - | ✓ | ✓ | - | - |
60| Text Extraction | - | ✓ | - | ✓ | - |
61| Structured Output | ✓ | ✓ | ✓ | ✓ | - |
62| Creation | TTS | - | - | - | ✓ |
63| Timestamps | ✓ | - | ✓ | - | - |
64| Segmentation | - | ✓ | - | - | - |
65
66## Model Selection Guide
67
68### Gemini 2.5 Series (Recommended)
69- **gemini-2.5-pro**: Highest quality, all features, 1M-2M context
70- **gemini-2.5-flash**: Best balance, all features, 1M-2M context
71- **gemini-2.5-flash-lite**: Lightweight, segmentation support
72- **gemini-2.5-flash-image**: Image generation only
73
74### Feature Requirements
75- **Segmentation**: Requires 2.5+ models
76- **Object Detection**: Requires 2.0+ models
77- **Multi-video**: Requires 2.5+ models
78- **Image Generation**: Requires flash-image model
79
80### Context Windows
81- **2M tokens**: ~6 hours video (low-res) or ~2 hours (default)
82- **1M tokens**: ~3 hours video (low-res) or ~1 hour (default)
83- **Audio**: 32 tokens/second (1 min = 1,920 tokens)
84- **PDF**: 258 tokens/page (fixed)
85- **Image**: 258-1,548 tokens based on size
86
87## Quick Start
88
89### Prerequisites
90
91**API Key Setup**: Supports both Google AI Studio and Vertex AI.
92
93The skill checks for `GEMINI_API_KEY` in this order:
941. Process environment: `export GEMINI_API_KEY="your-key"`
952. Project root: `.env`
963. `.claude/.env`
974. `.claude/skills/.env`
985. `.claude/skills/ai-multimodal/.env`
99
100**Get API key**: https://aistudio.google.com/apikey
101
102**For Vertex AI**:
103```bash
104export GEMINI_USE_VERTEX=true
105export VERTEX_PROJECT_ID=your-gcp-project-id
106export VERTEX_LOCATION=us-central1 # Optional
107```
108
109**Install SDK**:
110```bash
111pip install google-genai python-dotenv pillow
112```
113
114### Common Patterns
115
116**Transcribe Audio**:
117```bash
118python scripts/gemini_batch_process.py \
119 --files audio.mp3 \
120 --task transcribe \
121 --model gemini-2.5-flash
122```
123
124**Analyze Image**:
125```bash
126python scripts/gemini_batch_process.py \
127 --files image.jpg \
128 --task analyze \
129 --prompt "Describe this image" \
130 --output docs/assets/<output-name>.md \
131 --model gemini-2.5-flash
132```
133
134**Process Video**:
135```bash
136python scripts/gemini_batch_process.py \
137 --files video.mp4 \
138 --task analyze \
139 --prompt "Summarize key points with timestamps" \
140 --output docs/assets/<output-name>.md \
141 --model gemini-2.5-flash
142```
143
144**Extract from PDF**:
145```bash
146python scripts/gemini_batch_process.py \
147 --files document.pdf \
148 --task extract \
149 --prompt "Extract table data as JSON" \
150 --output docs/assets/<output-name>.md \
151 --format json
152```
153
154**Generate Image**:
155```bash
156python scripts/gemini_batch_process.py \
157 --task generate \
158 --prompt "A futuristic city at sunset" \
159 --output docs/assets/<output-file-name> \
160 --model gemini-2.5-flash-image \
161 --aspect-ratio 16:9
162```
163
164**Optimize Media**:
165```bash
166# Prepare large video for processing
167python scripts/media_optimizer.py \
168 --input large-video.mp4 \
169 --output docs/assets/<output-file-name> \
170 --target-size 100MB
171
172# Batch optimize multiple files
173python scripts/media_optimizer.py \
174 --input-dir ./videos \
175 --output-dir docs/assets/optimized \
176 --quality 85
177```
178
179**Convert Documents to Markdown**:
180```bash
181# Convert to PDF
182python scripts/document_converter.py \
183 --input document.docx \
184 --output docs/assets/document.md
185
186# Extract pages
187python scripts/document_converter.py \
188 --input large.pdf \
189 --output docs/assets/chapter1.md \
190 --pages 1-20
191```
192
193## Supported Formats
194
195### Audio
196- WAV, MP3, AAC, FLAC, OGG Vorbis, AIFF
197- Max 9.5 hours per request
198- Auto-downsampled to 16 Kbps mono
199
200### Images
201- PNG, JPEG, WEBP, HEIC, HEIF
202- Max 3,600 images per request
203- Resolution: ≤384px = 258 tokens, larger = tiled
204
205### Video
206- MP4, MPEG, MOV, AVI, FLV, MPG, WebM, WMV, 3GPP
207- Max 6 hours (low-res) or 2 hours (default)
208- YouTube URLs supported (public only)
209
210### Documents
211- PDF only for vision processing
212- Max 1,000 pages
213- TXT, HTML, Markdown supported (text-only)
214
215### Size Limits
216- **Inline**: <20MB total request
217- **File API**: 2GB per file, 20GB project quota
218- **Retention**: 48 hours auto-delete
219
220## Reference Navigation
221
222For detailed implementation guidance, see:
223
224### Audio Processing
225- `references/audio-processing.md` - Transcription, analysis, TTS
226 - Timestamp handling and segment analysis
227 - Multi-speaker identification
228 - Non-speech audio analysis
229 - Text-to-speech generation
230
231### Image Understanding
232- `references/vision-understanding.md` - Captioning, detection, OCR
233 - Object detection and localization
234 - Pixel-level segmentation
235 - Visual question answering
236 - Multi-image comparison
237
238### Video Analysis
239- `references/video-analysis.md` - Scene detection, temporal understanding
240 - YouTube URL processing
241 - Timestamp-based queries
242 - Video clipping and FPS control
243 - Long video optimization
244
245### Document Extraction
246- `references/document-extraction.md` - PDF processing, structured output
247 - Table and form extraction
248 - Chart and diagram analysis
249 - JSON schema validation
250 - Multi-page handling
251
252### Image Generation
253- `references/image-generation.md` - Text-to-image, editing
254 - Prompt engineering strategies
255 - Image editing and composition
256 - Aspect ratio selection
257 - Safety settings
258
259## Cost Optimization
260
261### Token Costs
262**Input Pricing**:
263- Gemini 2.5 Flash: $1.00/1M input, $0.10/1M output
264- Gemini 2.5 Pro: $3.00/1M input, $12.00/1M output
265- Gemini 1.5 Flash: $0.70/1M input, $0.175/1M output
266
267**Token Rates**:
268- Audio: 32 tokens/second (1 min = 1,920 tokens)
269- Video: ~300 tokens/second (default) or ~100 (low-res)
270- PDF: 258 tokens/page (fixed)
271- Image: 258-1,548 tokens based on size
272
273**TTS Pricing**:
274- Flash TTS: $10/1M tokens
275- Pro TTS: $20/1M tokens
276
277### Best Practices
2781. Use `gemini-2.5-flash` for most tasks (best price/performance)
2792. Use File API for files >20MB or repeated queries
2803. Optimize media before upload (see `media_optimizer.py`)
2814. Process specific segments instead of full videos
2825. Use lower FPS for static content
2836. Implement context caching for repeated queries
2847. Batch process multiple files in parallel
285
286## Rate Limits
287
288**Free Tier**:
289- 10-15 RPM (requests per minute)
290- 1M-4M TPM (tokens per minute)
291- 1,500 RPD (requests per day)
292
293**YouTube Limits**:
294- Free tier: 8 hours/day
295- Paid tier: No length limits
296- Public videos only
297
298**Storage Limits**:
299- 20GB per project
300- 2GB per file
301- 48-hour retention
302
303## Error Handling
304
305Common errors and solutions:
306- **400**: Invalid format/size - validate before upload
307- **401**: Invalid API key - check configuration
308- **403**: Permission denied - verify API key restrictions
309- **404**: File not found - ensure file uploaded and active
310- **429**: Rate limit exceeded - implement exponential backoff
311- **500**: Server error - retry with backoff
312
313## Scripts Overview
314
315All scripts support unified API key detection and error handling:
316
317**gemini_batch_process.py**: Batch process multiple media files
318- Supports all modalities (audio, image, video, PDF)
319- Progress tracking and error recovery
320- Output formats: JSON, Markdown, CSV
321- Rate limiting and retry logic
322- Dry-run mode
323
324**media_optimizer.py**: Prepare media for Gemini API
325- Compress videos/audio for size limits
326- Resize images appropriately
327- Split long videos into chunks
328- Format conversion
329- Quality vs size optimization
330
331**document_converter.py**: Convert documents to PDF
332- Convert DOCX, XLSX, PPTX to PDF
333- Extract page ranges
334- Optimize PDFs for Gemini
335- Extract images from PDFs
336- Batch conversion support
337
338Run any script with `--help` for detailed usage.
339
340## Resources
341
342- [Audio API Docs](https://ai.google.dev/gemini-api/docs/audio)
343- [Image API Docs](https://ai.google.dev/gemini-api/docs/image-understanding)
344- [Video API Docs](https://ai.google.dev/gemini-api/docs/video-understanding)
345- [Document API Docs](https://ai.google.dev/gemini-api/docs/document-processing)
346- [Image Gen Docs](https://ai.google.dev/gemini-api/docs/image-generation)
347- [Get API Key](https://aistudio.google.com/apikey)
348- [Pricing](https://ai.google.dev/pricing)
majiayu000