Vertex AI Media Master - Comprehensive Multimodal AI Operations
This Agent Skill provides comprehensive mastery of Google Vertex AI multimodal capabilities for video, audio, image, and text processing with focus on marketing applications.
Overview
The Vertex AI Media Master skill serves as a central orchestrator for all Google Cloud multimodal operations. It enables seamless integration between video processing, audio generation, image creation, and marketing campaign automation. The skill leverages Gemini 2.5 Pro's 2M context window to handle extensive media content while maintaining high-quality output across all modalities.
Core Capabilities
🎥 Video Processing (Gemini 2.0/2.5)
- Video Understanding: Process videos up to 6 hours at low resolution or 2 hours at default resolution
- 2M Context Window: Gemini 2.5 Pro handles massive video content
- Audio Track Processing: Automatic audio transcription from video
- Multi-video Analysis: Process multiple videos in single request
- Video Summarization: Extract key moments, scenes, and insights
- Marketing Use Cases:
- Analyze competitor video ads
- Extract highlights from long-form content
- Generate video summaries for social media
- Transcribe and caption video content
- Identify brand mentions and product placements
🎵 Audio Generation & Processing
- Lyria Model (2025): Native audio and music generation
- Speech-to-Text: Transcribe audio with speaker diarization
- Text-to-Speech: Generate natural voiceovers
- Music Composition: Background music for campaigns
- Audio Enhancement: Noise reduction and quality improvement
- Marketing Use Cases:
- Generate podcast scripts and voiceovers
- Create audio ads and radio spots
- Produce background music for video campaigns
- Transcribe customer interviews
- Generate multilingual voiceovers
🖼️ Image Generation (Imagen 4 & Gemini 2.5 Flash Image)
- Imagen 4: Highest quality text-to-image generation
- Gemini 2.5 Flash Image: Interleaved image generation with text
- Style Transfer: Apply brand styles to generated images
- Product Visualization: Generate product mockups
- Campaign Assets: Create ad creatives and social media graphics
- Marketing Use Cases:
- Generate personalized ad images (Adios solution)
- Create social media graphics at scale
- Produce product lifestyle images
- Generate A/B test variations
- Create branded campaign visuals
📢 Marketing Campaign Automation
- ViGenAiR: Convert long-form video ads to short formats automatically
- Adios: Generate personalized ad images tailored to audience context
- Campaign Asset Generation: Photos, soundtracks, voiceovers from prompts
- Content Pipeline: Email copy, blog posts, social media, PMax assets
- Catalog Enrichment: Multi-agent workflow for product onboarding
- Marketing Use Cases:
- Automated campaign asset production
- Personalized content at scale
- Multi-channel content distribution
- Product catalog enhancement
- Visual merchandising automation
🔧 Technical Implementation
API Integration:
from google.cloud import aiplatform
from vertexai.preview.generative_models import GenerativeModel
# Initialize Vertex AI
aiplatform.init(project="your-project", location="us-central1")
# Gemini 2.5 Pro for video
model = GenerativeModel("gemini-2.5-pro")
# Process video with audio
response = model.generate_content([
"Analyze this video and extract key marketing insights",
video_file, # Up to 6 hours
])
# Imagen 4 for image generation
from vertexai.preview.vision_models import ImageGenerationModel
imagen = ImageGenerationModel.from_pretrained("imagen-4")
images = imagen.generate_images(
prompt="Professional product photo, studio lighting, white background",
number_of_images=4
)
Gemini 2.5 Flash Image (Interleaved Generation):
# Generate images within text responses
model = GenerativeModel("gemini-2.5-flash-image")
response = model.generate_content([
"Create a 5-step recipe with images for each step"
])
# Returns text + images interleaved
Audio Generation (Lyria):
from vertexai.preview.audio_models import AudioGenerationModel
lyria = AudioGenerationModel.from_pretrained("lyria")
audio = lyria.generate_audio(
prompt="Upbeat background music for product launch video, 30 seconds",
duration=30
)
📊 Marketing Workflow Automation
1. Multi-Channel Campaign Creation:
# Single prompt generates all assets
campaign = model.generate_content([
"""Create a product launch campaign for [product]:
- Hero image (1920x1080)
- 3 social media graphics (1080x1080)
- 30-second video script
- Background music description
- Email marketing copy
- Instagram caption"""
])
2. Video Repurposing Pipeline:
# Long-form to short-form conversion (ViGenAiR approach)
long_video = "gs://bucket/original-ad-60s.mp4"
response = model.generate_content([
f"Extract 3 engaging 15-second clips from this video for TikTok/Reels",
long_video
])
# Auto-generates format-specific versions
3. Personalized Ad Generation:
# Context-aware image generation (Adios approach)
for audience in audiences:
ad_image = imagen.generate_images(
prompt=f"Product ad for {product}, targeting {audience.demographics}, {audience.style_preference}",
aspect_ratio="16:9"
)
🎯 Best Practices for Jeremy
1. Project Setup:
# Set environment variables
export GOOGLE_CLOUD_PROJECT="your-project-id"
export GOOGLE_APPLICATION_CREDENTIALS="path/to/service-account.json"
# Install SDK
pip install google-cloud-aiplatform[vision,audio] google-generativeai
2. Rate Limits & Quotas:
- Gemini 2.5 Pro: 2M tokens/min (video processing)
- Imagen 4: 100 images/min
- Monitor usage in Cloud Console
3. Cost Optimization:
- Use Gemini 2.5 Flash for faster, cheaper operations
- Batch image generation requests
- Cache video embeddings for repeated analysis
- Use low-resolution video setting when appropriate
4. Security & Compliance:
- Keep API keys in Secret Manager, never in code
- Use service accounts with minimal permissions
- Enable VPC Service Controls for data residency
- Log all API calls for audit trails
🚀 Advanced Marketing Use Cases
1. Campaign Performance Analysis:
# Analyze competitor campaigns
competitor_videos = ["gs://bucket/competitor1.mp4", "gs://bucket/competitor2.mp4"]
analysis = model.generate_content([
"Compare these competitor videos: themes, messaging, CTAs, production quality",
*competitor_videos
])
2. Content Localization:
# Generate multilingual campaigns
for lang in ["en", "es", "fr", "de", "ja"]:
localized_content = model.generate_content([
f"Translate and culturally adapt this campaign for {lang} market:",
campaign_brief,
hero_image
])
3. A/B Test Generation:
# Generate variations automatically
variations = []
for style in ["minimalist", "bold", "luxury", "playful"]:
variation = imagen.generate_images(
prompt=f"Product ad, {style} style, {brand_guidelines}",
number_of_images=1
)
variations.append(variation)
📚 Reference Documentation
Official Documentation:
- Vertex AI Multimodal: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/overview
- Gemini 2.5 Pro: https://cloud.google.com/vertex-ai/generative-ai/docs/models
- Imagen 4: https://cloud.google.com/vertex-ai/generative-ai/docs/image/overview
- Video Understanding: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/video-understanding
Marketing Solutions:
- GenAI for Marketing: https://github.com/GoogleCloudPlatform/genai-for-marketing
- ViGenAiR (video repurposing)
- Adios (personalized ad images)
Pricing:
- Gemini 2.5 Pro: $3.50/1M input tokens, $10.50/1M output tokens
- Imagen 4: $0.04/image
- Video processing: Included in Gemini token pricing
Guidelines
This section outlines operational procedures and configuration guidelines for optimal skill performance.
Operational Guidelines
- System Initialization: Ensure all Vertex AI services are properly configured before initiating media operations.
- Resource Allocation: Monitor token consumption and adjust quotas based on workload demands.
- Quality Assurance: Implement automated testing for all media generation workflows.
- Performance Monitoring: Track key metrics including response times, error rates, and throughput.
Runtime Configuration
The following configuration parameters should be maintained for consistent performance:
- API Endpoints: Maintain connectivity to all Vertex AI service endpoints
- Authentication: Utilize service account credentials for secure API access
- Caching Strategy: Implement caching for frequently accessed media assets
- Error Handling: Establish robust error recovery mechanisms for all operations
System Instructions
Configuration Parameters
| Parameter | Value | Description |
|---|---|---|
| Max Video Duration | 6 hours | Maximum supported video length |
| Token Limit | 2M | Context window capacity |
| Image Resolution | 1920x1080 | Standard output resolution |
| Audio Format | MP3/AAC | Supported audio codecs |
Deployment Checklist
- Verify service account permissions
- Configure network security settings
- Set up monitoring and alerting
- Document API integration points
- Establish backup and recovery procedures
Examples
Example 1: Video Analysis Workflow
# Process a marketing video and extract insights
video_analysis = model.generate_content([
"Analyze this video for brand messaging and engagement opportunities",
video_file
])
Example 2: Multi-Modal Content Generation
# Generate comprehensive marketing content
content = model.generate_content([
"Create a complete marketing package including video, images, and audio"
])
Example 3: Campaign Asset Production
# Produce campaign assets for multiple channels
assets = model.generate_content([
"Generate assets for social media, email, and web platforms"
])
When This Skill Activates
This skill automatically activates when you mention:
- Video processing, analysis, or understanding
- Audio generation, music composition, or voiceovers
- Image generation, ad creatives, or visual content
- Marketing campaigns, content automation, or asset production
- Gemini multimodal capabilities
- Vertex AI media operations
- Social media content, email marketing, or PMax campaigns
Integration with Other Tools
Google Cloud Services:
- Cloud Storage for media asset management
- BigQuery for campaign analytics
- Cloud Functions for automation triggers
- Vertex AI Pipelines for content workflows
Third-Party Integrations:
- Social media APIs (LinkedIn, Twitter, Instagram)
- Marketing automation platforms (HubSpot, Marketo)
- CMS integrations (WordPress, Contentful)
- DAM systems (Bynder, Cloudinary)
Success Metrics
Track These KPIs:
- Asset generation speed (baseline: 5 images/min)
- Content approval rate (target: >80%)
- Campaign personalization scale (target: 1000+ variants)
- Cost per asset (target: <$0.10/image)
- Time saved vs manual production (target: 90% reduction)
This skill makes Jeremy a Vertex AI multimodal expert with instant access to video processing, audio generation, image creation, and marketing automation capabilities.