AI Multimodal Processing Skill
Process audio, images, videos, documents, and generate images using Google Gemini's multimodal API. Unified interface for all multimedia content understanding and generation.
Core Capabilities
Audio Processing
- Transcription with timestamps (up to 9.5 hours)
- Audio summarization and analysis
- Speech understanding and speaker identification
- Music and environmental sound analysis
- Text-to-speech generation with controllable voice
Image Understanding
- Image captioning and description
- Object detection with bounding boxes (2.0+)
- Pixel-level segmentation (2.5+)
- Visual question answering
- Multi-image comparison (up to 3,600 images)
- OCR and text extraction
Video Analysis
- Scene detection and summarization
- Video Q&A with temporal understanding
- Transcription with visual descriptions
- YouTube URL support
- Long video processing (up to 6 hours)
- Frame-level analysis
Document Extraction
- Native PDF vision processing (up to 1,000 pages)
- Table and form extraction
- Chart and diagram analysis
- Multi-page document understanding
- Structured data output (JSON schema)
- Format conversion (PDF to HTML/JSON)
Image Generation
- Text-to-image generation
- Image editing and modification
- Multi-image composition (up to 3 images)
- Iterative refinement
- Multiple aspect ratios (1:1, 16:9, 9:16, 4:3, 3:4)
- Controllable style and quality
Capability Matrix
| Task |
Audio |
Image |
Video |
Document |
Generation |
| Transcription |
Y |
- |
Y |
- |
- |
| Summarization |
Y |
Y |
Y |
Y |
- |
| Q&A |
Y |
Y |
Y |
Y |
- |
| Object Detection |
- |
Y |
Y |
- |
- |
| Text Extraction |
- |
Y |
- |
Y |
- |
| Structured Output |
Y |
Y |
Y |
Y |
- |
| Creation |
TTS |
- |
- |
- |
Y |
| Timestamps |
Y |
- |
Y |
- |
- |
| Segmentation |
- |
Y |
- |
- |
- |
Model Selection Guide
Gemini 2.5 Series (Recommended)
- gemini-2.5-pro: Highest quality, all features, 1M-2M context
- gemini-2.5-flash: Best balance, all features, 1M-2M context
- gemini-2.5-flash-lite: Lightweight, segmentation support
- gemini-2.5-flash-image: Image generation only
Gemini 2.0 Series
- gemini-2.0-flash: Fast processing, object detection
- gemini-2.0-flash-lite: Lightweight option
Feature Requirements
- Segmentation: Requires 2.5+ models
- Object Detection: Requires 2.0+ models
- Multi-video: Requires 2.5+ models
- Image Generation: Requires flash-image model
Context Windows
- 2M tokens: ~6 hours video (low-res) or ~2 hours (default)
- 1M tokens: ~3 hours video (low-res) or ~1 hour (default)
- Audio: 32 tokens/second (1 min = 1,920 tokens)
- PDF: 258 tokens/page (fixed)
- Image: 258-1,548 tokens based on size
Quick Start
Prerequisites
API Key Setup: Supports both Google AI Studio and Vertex AI.
The skill checks for GEMINI_API_KEY in this order:
- Process environment:
export GEMINI_API_KEY="your-key"
- Project root:
.env
.claude/.env.claude/skills/.env.claude/skills/ai-multimodal/.env
Get API key: https://aistudio.google.com/apikey
For Vertex AI:
Install SDK:
Common Patterns
Transcribe Audio:
Analyze Image:
Process Video:
Extract from PDF:
Generate Image:
Optimize Media:
Batch optimize multiple files
python scripts/media_optimizer.py
--input-dir ./videos
--output-dir docs/assets/optimized
--quality 85
Convert Documents to Markdown:
Extract pages
python scripts/document_converter.py
--input large.pdf
--output docs/assets/chapter1.md
--pages 1-20
Supported Formats
Audio
- WAV, MP3, AAC, FLAC, OGG Vorbis, AIFF
- Max 9.5 hours per request
- Auto-downsampled to 16 Kbps mono
Images
- PNG, JPEG, WEBP, HEIC, HEIF
- Max 3,600 images per request
- Resolution: <=384px = 258 tokens, larger = tiled
Video
- MP4, MPEG, MOV, AVI, FLV, MPG, WebM, WMV, 3GPP
- Max 6 hours (low-res) or 2 hours (default)
- YouTube URLs supported (public only)
Documents
- PDF only for vision processing
- Max 1,000 pages
- TXT, HTML, Markdown supported (text-only)
Reference Navigation
For detailed implementation guidance, see:
Audio Processing
references/audio-processing.md - Transcription, analysis, TTS
- Timestamp handling and segment analysis
- Multi-speaker identification
- Non-speech audio analysis
- Text-to-speech generation
Image Understanding
references/vision-understanding.md - Captioning, detection, OCR
- Object detection and localization
- Pixel-level segmentation
- Visual question answering
- Multi-image comparison
Video Analysis
references/video-analysis.md - Scene detection, temporal understanding
- YouTube URL processing
- Timestamp-based queries
- Video clipping and FPS control
- Long video optimization
Document Extraction
references/document-extraction.md - PDF processing, structured output
- Table and form extraction
- Chart and diagram analysis
- JSON schema validation
- Multi-page handling
Image Generation
references/image-generation.md - Text-to-image, editing
- Prompt engineering strategies
- Image editing and composition
- Aspect ratio selection
- Safety settings
Cost Optimization
Token Costs
Input Pricing:
- Gemini 2.5 Flash: $1.00/1M input, $0.10/1M output
- Gemini 2.5 Pro: $3.00/1M input, $12.00/1M output
- Gemini 1.5 Flash: $0.70/1M input, $0.175/1M output
Token Rates:
- Audio: 32 tokens/second (1 min = 1,920 tokens)
- Video: ~300 tokens/second (default) or ~100 (low-res)
- PDF: 258 tokens/page (fixed)
- Image: 258-1,548 tokens based on size
TTS Pricing:
- Flash TTS: $10/1M tokens
- Pro TTS: $20/1M tokens
Best Practices
- Use
gemini-2.5-flash for most tasks (best price/performance)
- Use File API for files >20MB or repeated queries
- Optimize media before upload (see
media_optimizer.py)
- Process specific segments instead of full videos
- Use lower FPS for static content
- Implement context caching for repeated queries
- Batch process multiple files in parallel
Rate Limits
Free Tier:
- 10-15 RPM (requests per minute)
- 1M-4M TPM (tokens per minute)
- 1,500 RPD (requests per day)
YouTube Limits:
- Free tier: 8 hours/day
- Paid tier: No length limits
- Public videos only
Storage Limits:
- 20GB per project
- 2GB per file
- 48-hour retention
Error Handling
Common errors and solutions:
- 400: Invalid format/size - validate before upload
- 401: Invalid API key - check configuration
- 403: Permission denied - verify API key restrictions
- 404: File not found - ensure file uploaded and active
- 429: Rate limit exceeded - implement exponential backoff
- 500: Server error - retry with backoff
Scripts Overview
All scripts support unified API key detection and error handling:
gemini_batch_process.py: Batch process multiple media files
- Supports all modalities (audio, image, video, PDF)
- Progress tracking and error recovery
- Output formats: JSON, Markdown, CSV
- Rate limiting and retry logic
- Dry-run mode
media_optimizer.py: Prepare media for Gemini API
- Compress videos/audio for size limits
- Resize images appropriately
- Split long videos into chunks
- Format conversion
- Quality vs size optimization
document_converter.py: Convert documents to PDF
- Convert DOCX, XLSX, PPTX to PDF
- Extract page ranges
- Optimize PDFs for Gemini
- Extract images from PDFs
- Batch conversion support
Run any script with --help for detailed usage.
Resources
1---2name: ai-multimodal3description: Process and generate multimedia content using Google Gemini API. Capabilities include analyze audio files (transcription with timestamps, summarization, speech understanding, music/sound analysis up to 9.5 hours), understand images (captioning, object detection, OCR, visual Q&A, segmentation), process videos (scene detection, Q&A, temporal analysis, YouTube URLs, up to 6 hours), extract from documents (PDF tables, forms, charts, diagrams, multi-page), generate images (text-to-image, editing, composition, refinement). Use when working with audio/video files, analyzing images or screenshots, processing PDF documents, extracting structured data from media, creating images from text prompts, or implementing multimodal AI features. Supports multiple models (Gemini 2.5/2.0) with context windows up to 2M tokens.4license: MIT5---67# AI Multimodal Processing Skill89Process audio, images, videos, documents, and generate images using Google Gemini's multimodal API. Unified interface for all multimedia content understanding and generation.1011## Core Capabilities1213### Audio Processing14- Transcription with timestamps (up to 9.5 hours)15- Audio summarization and analysis16- Speech understanding and speaker identification17- Music and environmental sound analysis18- Text-to-speech generation with controllable voice1920### Image Understanding21- Image captioning and description22- Object detection with bounding boxes (2.0+)23- Pixel-level segmentation (2.5+)24- Visual question answering25- Multi-image comparison (up to 3,600 images)26- OCR and text extraction2728### Video Analysis29- Scene detection and summarization30- Video Q&A with temporal understanding31- Transcription with visual descriptions32- YouTube URL support33- Long video processing (up to 6 hours)34- Frame-level analysis3536### Document Extraction37- Native PDF vision processing (up to 1,000 pages)38- Table and form extraction39- Chart and diagram analysis40- Multi-page document understanding41- Structured data output (JSON schema)42- Format conversion (PDF to HTML/JSON)4344### Image Generation45- Text-to-image generation46- Image editing and modification47- Multi-image composition (up to 3 images)48- Iterative refinement49- Multiple aspect ratios (1:1, 16:9, 9:16, 4:3, 3:4)50- Controllable style and quality5152## Capability Matrix5354| Task | Audio | Image | Video | Document | Generation |55|------|:-----:|:-----:|:-----:|:--------:|:----------:|56| Transcription | Y | - | Y | - | - |57| Summarization | Y | Y | Y | Y | - |58| Q&A | Y | Y | Y | Y | - |59| Object Detection | - | Y | Y | - | - |60| Text Extraction | - | Y | - | Y | - |61| Structured Output | Y | Y | Y | Y | - |62| Creation | TTS | - | - | - | Y |63| Timestamps | Y | - | Y | - | - |64| Segmentation | - | Y | - | - | - |6566## Model Selection Guide6768### Gemini 2.5 Series (Recommended)69- **gemini-2.5-pro**: Highest quality, all features, 1M-2M context70- **gemini-2.5-flash**: Best balance, all features, 1M-2M context71- **gemini-2.5-flash-lite**: Lightweight, segmentation support72- **gemini-2.5-flash-image**: Image generation only7374### Gemini 2.0 Series75- **gemini-2.0-flash**: Fast processing, object detection76- **gemini-2.0-flash-lite**: Lightweight option7778### Feature Requirements79- **Segmentation**: Requires 2.5+ models80- **Object Detection**: Requires 2.0+ models81- **Multi-video**: Requires 2.5+ models82- **Image Generation**: Requires flash-image model8384### Context Windows85- **2M tokens**: ~6 hours video (low-res) or ~2 hours (default)86- **1M tokens**: ~3 hours video (low-res) or ~1 hour (default)87- **Audio**: 32 tokens/second (1 min = 1,920 tokens)88- **PDF**: 258 tokens/page (fixed)89- **Image**: 258-1,548 tokens based on size9091## Quick Start9293### Prerequisites9495**API Key Setup**: Supports both Google AI Studio and Vertex AI.9697The skill checks for `GEMINI_API_KEY` in this order:981. Process environment: `export GEMINI_API_KEY="your-key"`992. Project root: `.env`1003. `.claude/.env`1014. `.claude/skills/.env`1025. `.claude/skills/ai-multimodal/.env`103104**Get API key**: https://aistudio.google.com/apikey105106**For Vertex AI**:107108<example type="usage">109<code language="bash">110export GEMINI_USE_VERTEX=true111export VERTEX_PROJECT_ID=your-gcp-project-id112export VERTEX_LOCATION=us-central1 # Optional113</code>114</example>115116**Install SDK**:117118<example type="usage">119<code language="bash">120pip install google-genai python-dotenv pillow121</code>122</example>123124### Common Patterns125126**Transcribe Audio**:127128<example type="usage">129<code language="bash">130python scripts/gemini_batch_process.py \131 --files audio.mp3 \132 --task transcribe \133 --model gemini-2.5-flash134</code>135</example>136137**Analyze Image**:138139<example type="usage">140<code language="bash">141python scripts/gemini_batch_process.py \142 --files image.jpg \143 --task analyze \144 --prompt "Describe this image" \145 --output docs/assets/<output-name>.md \146 --model gemini-2.5-flash147</code>148</example>149150**Process Video**:151152<example type="usage">153<code language="bash">154python scripts/gemini_batch_process.py \155 --files video.mp4 \156 --task analyze \157 --prompt "Summarize key points with timestamps" \158 --output docs/assets/<output-name>.md \159 --model gemini-2.5-flash160</code>161</example>162163**Extract from PDF**:164165<example type="usage">166<code language="bash">167python scripts/gemini_batch_process.py \168 --files document.pdf \169 --task extract \170 --prompt "Extract table data as JSON" \171 --output docs/assets/<output-name>.md \172 --format json173</code>174</example>175176**Generate Image**:177178<example type="usage">179<code language="bash">180python scripts/gemini_batch_process.py \181 --task generate \182 --prompt "A futuristic city at sunset" \183 --output docs/assets/<output-file-name> \184 --model gemini-2.5-flash-image \185 --aspect-ratio 16:9186</code>187</example>188189**Optimize Media**:190191<example type="usage">192<code language="bash">193# Prepare large video for processing194python scripts/media_optimizer.py \195 --input large-video.mp4 \196 --output docs/assets/<output-file-name> \197 --target-size 100MB198199# Batch optimize multiple files200python scripts/media_optimizer.py \201 --input-dir ./videos \202 --output-dir docs/assets/optimized \203 --quality 85204</code>205</example>206207**Convert Documents to Markdown**:208209<example type="usage">210<code language="bash">211# Convert to PDF212python scripts/document_converter.py \213 --input document.docx \214 --output docs/assets/document.md215216# Extract pages217python scripts/document_converter.py \218 --input large.pdf \219 --output docs/assets/chapter1.md \220 --pages 1-20221</code>222</example>223224## Supported Formats225226### Audio227- WAV, MP3, AAC, FLAC, OGG Vorbis, AIFF228- Max 9.5 hours per request229- Auto-downsampled to 16 Kbps mono230231### Images232- PNG, JPEG, WEBP, HEIC, HEIF233- Max 3,600 images per request234- Resolution: <=384px = 258 tokens, larger = tiled235236### Video237- MP4, MPEG, MOV, AVI, FLV, MPG, WebM, WMV, 3GPP238- Max 6 hours (low-res) or 2 hours (default)239- YouTube URLs supported (public only)240241### Documents242- PDF only for vision processing243- Max 1,000 pages244- TXT, HTML, Markdown supported (text-only)245246<constraints>247<constraint severity="critical">API key must be kept secure - never commit to version control</constraint>248<constraint severity="high">File size limit: 20MB inline, 2GB via File API</constraint>249<constraint severity="high">YouTube processing limited to public videos only</constraint>250<constraint severity="medium">Free tier rate limit: 10-15 requests per minute</constraint>251<constraint severity="medium">Files uploaded via File API are auto-deleted after 48 hours</constraint>252<constraint severity="low">Image generation requires specific flash-image model</constraint>253</constraints>254255## Reference Navigation256257For detailed implementation guidance, see:258259### Audio Processing260- `references/audio-processing.md` - Transcription, analysis, TTS261 - Timestamp handling and segment analysis262 - Multi-speaker identification263 - Non-speech audio analysis264 - Text-to-speech generation265266### Image Understanding267- `references/vision-understanding.md` - Captioning, detection, OCR268 - Object detection and localization269 - Pixel-level segmentation270 - Visual question answering271 - Multi-image comparison272273### Video Analysis274- `references/video-analysis.md` - Scene detection, temporal understanding275 - YouTube URL processing276 - Timestamp-based queries277 - Video clipping and FPS control278 - Long video optimization279280### Document Extraction281- `references/document-extraction.md` - PDF processing, structured output282 - Table and form extraction283 - Chart and diagram analysis284 - JSON schema validation285 - Multi-page handling286287### Image Generation288- `references/image-generation.md` - Text-to-image, editing289 - Prompt engineering strategies290 - Image editing and composition291 - Aspect ratio selection292 - Safety settings293294## Cost Optimization295296### Token Costs297**Input Pricing**:298- Gemini 2.5 Flash: $1.00/1M input, $0.10/1M output299- Gemini 2.5 Pro: $3.00/1M input, $12.00/1M output300- Gemini 1.5 Flash: $0.70/1M input, $0.175/1M output301302**Token Rates**:303- Audio: 32 tokens/second (1 min = 1,920 tokens)304- Video: ~300 tokens/second (default) or ~100 (low-res)305- PDF: 258 tokens/page (fixed)306- Image: 258-1,548 tokens based on size307308**TTS Pricing**:309- Flash TTS: $10/1M tokens310- Pro TTS: $20/1M tokens311312### Best Practices3131. Use `gemini-2.5-flash` for most tasks (best price/performance)3142. Use File API for files >20MB or repeated queries3153. Optimize media before upload (see `media_optimizer.py`)3164. Process specific segments instead of full videos3175. Use lower FPS for static content3186. Implement context caching for repeated queries3197. Batch process multiple files in parallel320321## Rate Limits322323**Free Tier**:324- 10-15 RPM (requests per minute)325- 1M-4M TPM (tokens per minute)326- 1,500 RPD (requests per day)327328**YouTube Limits**:329- Free tier: 8 hours/day330- Paid tier: No length limits331- Public videos only332333**Storage Limits**:334- 20GB per project335- 2GB per file336- 48-hour retention337338## Error Handling339340Common errors and solutions:341- **400**: Invalid format/size - validate before upload342- **401**: Invalid API key - check configuration343- **403**: Permission denied - verify API key restrictions344- **404**: File not found - ensure file uploaded and active345- **429**: Rate limit exceeded - implement exponential backoff346- **500**: Server error - retry with backoff347348## Scripts Overview349350All scripts support unified API key detection and error handling:351352**gemini_batch_process.py**: Batch process multiple media files353- Supports all modalities (audio, image, video, PDF)354- Progress tracking and error recovery355- Output formats: JSON, Markdown, CSV356- Rate limiting and retry logic357- Dry-run mode358359**media_optimizer.py**: Prepare media for Gemini API360- Compress videos/audio for size limits361- Resize images appropriately362- Split long videos into chunks363- Format conversion364- Quality vs size optimization365366**document_converter.py**: Convert documents to PDF367- Convert DOCX, XLSX, PPTX to PDF368- Extract page ranges369- Optimize PDFs for Gemini370- Extract images from PDFs371- Batch conversion support372373Run any script with `--help` for detailed usage.374375## Resources376377- [Audio API Docs](https://ai.google.dev/gemini-api/docs/audio)378- [Image API Docs](https://ai.google.dev/gemini-api/docs/image-understanding)379- [Video API Docs](https://ai.google.dev/gemini-api/docs/video-understanding)380- [Document API Docs](https://ai.google.dev/gemini-api/docs/document-processing)381- [Image Gen Docs](https://ai.google.dev/gemini-api/docs/image-generation)382- [Get API Key](https://aistudio.google.com/apikey)383- [Pricing](https://ai.google.dev/pricing)
zircote