Multimodal LLM Patterns
Integrate vision, audio, and video generation capabilities from leading multimodal models. Covers image analysis, document understanding, real-time voice agents, speech-to-text, text-to-speech, and AI video generation (Kling v3, Sora 2, Veo 3.1 std/lite/fast tiers, Runway Gen-4.5 via gen4_turbo).
Canonical model IDs (Google IDs checked against ai.google.dev/gemini-api/docs/models on 2026-09-23):
| Provider |
Model IDs |
| Anthropic |
claude-opus-5-5 (recommended, the default Opus since CC 2.1.280, same 2,576 px budget as Opus 5), claude-opus-5, claude-opus-4-8, claude-opus-4-7, claude-opus-4-6, claude-sonnet-5-5, claude-sonnet-4-6, claude-haiku-4-5-20251001. claude-fable-5 is Anthropic's frontier tier above Opus (GA 2026-07). Premium cost, never auto-pin it; the fable-spend-consent gate requires explicit user consent before any Fable spend |
| OpenAI |
gpt-5.5 (current flagship) |
| Google |
gemini-3.1-pro-preview (flagship), gemini-3.1-flash-lite (cost) |
| Veo |
veo-3.1-generate-preview / veo-3.1-lite-generate-preview / veo-3.1-fast-generate-preview |
| Kling |
kling-v3 (model_name field in Kling API) |
| Runway |
gen4_turbo (product label: Gen-4.5) |
Quick Reference
| Category |
Rules |
Impact |
When to Use |
| Vision: Image Analysis |
1 |
HIGH |
Image captioning, VQA, multi-image comparison, object detection |
| Vision: Document Understanding |
1 |
HIGH |
OCR, chart/diagram analysis, PDF processing, table extraction |
| Vision: Model Selection |
1 |
MEDIUM |
Choosing provider, cost optimization, image size limits |
| Audio: Speech-to-Text |
1 |
HIGH |
Transcription, speaker diarization, long-form audio |
| Audio: Text-to-Speech |
1 |
MEDIUM |
Voice synthesis, expressive TTS, multi-speaker dialogue |
| Audio: Model Selection |
1 |
MEDIUM |
Real-time voice agents, provider comparison, pricing |
| Video: Model Selection |
1 |
HIGH |
Choosing video gen provider (Kling, Sora, Veo, Runway) |
| Video: API Patterns |
1 |
HIGH |
Async task polling, SDK integration, webhook callbacks |
| Video: Multi-Shot |
1 |
HIGH |
Storyboarding, character elements, scene consistency |
Total: 9 rules across 3 categories (Vision, Audio, Video Generation)
Vision: Image Analysis
Send images to multimodal LLMs for captioning, visual QA, and object detection. Always set max_tokens and resize images before encoding.
| Rule |
File |
Key Pattern |
| Image Analysis |
rules/vision-image-analysis.md |
Base64 encoding, multi-image, bounding boxes |
Vision: Document Understanding
Extract structured data from documents, charts, and PDFs using vision models.
| Rule |
File |
Key Pattern |
| Document Vision |
rules/vision-document.md |
PDF page ranges, detail levels, OCR strategies |
Vision: Model Selection
Choose the right vision provider based on accuracy, cost, and context window needs.
| Rule |
File |
Key Pattern |
| Vision Models |
rules/vision-models.md |
Provider comparison, token costs, image limits |
Audio: Speech-to-Text
Convert audio to text with speaker diarization, timestamps, and sentiment analysis.
| Rule |
File |
Key Pattern |
| Speech-to-Text |
rules/audio-speech-to-text.md |
Gemini long-form, GPT-4o-Transcribe, AssemblyAI features |
Audio: Text-to-Speech
Generate natural speech from text with voice selection and expressive cues.
| Rule |
File |
Key Pattern |
| Text-to-Speech |
rules/audio-text-to-speech.md |
Gemini TTS, voice config, auditory cues |
Audio: Model Selection
Select the right audio/voice provider for real-time, transcription, or TTS use cases.
| Rule |
File |
Key Pattern |
| Audio Models |
rules/audio-models.md |
Real-time voice comparison, STT benchmarks, pricing |
Video: Model Selection
Choose the right video generation provider based on use case, duration, and budget.
| Rule |
File |
Key Pattern |
| Video Models |
rules/video-generation-models.md |
Kling vs Sora vs Veo vs Runway, pricing, capabilities |
Video: API Patterns
Integrate video generation APIs with proper async polling, SDKs, and webhook callbacks.
| Rule |
File |
Key Pattern |
| API Integration |
rules/video-generation-patterns.md |
Kling REST, fal.ai SDK, Vercel AI SDK, task polling |
Video: Multi-Shot
Generate multi-scene videos with consistent characters using storyboarding and character elements.
| Rule |
File |
Key Pattern |
| Multi-Shot |
rules/video-multi-shot.md |
Kling v3 character elements, 6-shot storyboards, identity binding |
Key Decisions
| Decision |
Recommendation |
| High accuracy vision |
claude-opus-5-5 (default Opus since CC 2.1.280, 2,576 px vision budget, 3× what Opus 4.6 allotted; give it crop/analyze tools rather than more thinking, which is the cheaper lever on this model). (claude-fable-5 is the frontier SOTA option, GA 2026-07 — premium cost, use only with explicit consent via the fable-spend-consent gate) |
| Long documents |
gemini-3.1-pro-preview (1M+ context) |
| Cost-efficient vision |
gemini-3.1-flash-lite (GA successor of the Flash-Lite preview, which was shut down 2026-05-25; shutdown scheduled 2027-05-07) |
| Video analysis |
gemini-3.1-pro-preview (native video, supersedes 2.5 Pro) |
| Voice assistant |
Grok Voice Agent on Grok 4.20 (fastest, <1s) |
| Emotional voice AI |
Gemini Live API |
| Long audio transcription |
gemini-3.1-pro-preview (9.5hr) |
| Speaker diarization |
AssemblyAI or Gemini |
| Self-hosted STT |
Whisper Large V3 |
| Character-consistent video |
kling-v3 (Character Elements 3.0) |
| Narrative video / storytelling |
Sora 2 (best cause-and-effect coherence) |
| Cinematic B-roll |
veo-3.1-generate-preview (camera control + polished motion) |
| Budget drafts |
veo-3.1-lite-generate-preview (~$0.05/s, 720/1080p) |
| Mid-tier fast renders |
veo-3.1-fast-generate-preview |
| Professional VFX |
Runway gen4_turbo (Act-Two motion transfer) |
| High-volume social video |
kling-v3 Standard (~$0.20/video) |
| Open-source video gen |
Wan 2.6 or LTX-2 |
| Lip-sync / avatar video |
kling-v3 (native lip-sync API) |
Example
import anthropic, base64
client = anthropic.Anthropic()
with open("image.png", "rb") as f:
b64 = base64.standard_b64encode(f.read()).decode("utf-8")
response = client.messages.create(
model="claude-opus-5-5",
max_tokens=1024,
messages=[{"role": "user", "content": [
{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
{"type": "text", "text": "Describe this image"}
]}]
)
Common Mistakes
- Not setting
max_tokens on vision requests (responses truncated)
- Sending oversized images without resizing (>2048px)
- Using
high detail level for simple yes/no classification
- Using STT+LLM+TTS pipeline instead of native speech-to-speech
- Not leveraging barge-in support for natural voice conversations
- Using deprecated models (GPT-4V, Whisper-1)
- Ignoring rate limits on vision and audio endpoints
- Calling video generation APIs synchronously (they're async — poll or use callbacks)
- Generating separate clips without character elements (characters look different each time)
- Using Sora for high-volume social content (expensive, slow — use Kling Standard instead)
Related Skills
ork-extras:rag-retrieval - Multimodal RAG with image + text retrieval
ork-extras:llm-integration - General LLM function calling patterns
streaming-api-patterns - WebSocket patterns for real-time audio
ork:demo-producer - Terminal demo videos (VHS, asciinema) — not AI video gen
1---2name: multimodal-llm3license: MIT4compatibility: "Claude Code 2.1.277+."5description: Vision, audio, video generation, and multimodal LLM patterns. Use when processing images, transcribing audio, generating speech, generating AI video (Kling v3, Sora 2, Veo 3.1, Runway Gen-4.5), or building multimodal AI pipelines.6user-invocable: false7effort: high8metadata:9 source: https://skillmd.com/skills/yonatangross/multimodal-llm10 owner-agent: "multimodal-specialist"11 category: mcp-enhancement12 version: "2.1.1"13 author: "OrchestKit"14 complexity: "high"15 tags: "vision, audio, video, multimodal, image, speech, transcription, tts, kling, sora, veo, video-generation"16allowed-tools: "Read Glob Grep WebFetch WebSearch"17---1819# Multimodal LLM Patterns2021Integrate vision, audio, and video generation capabilities from leading multimodal models. Covers image analysis, document understanding, real-time voice agents, speech-to-text, text-to-speech, and AI video generation (Kling v3, Sora 2, Veo 3.1 std/lite/fast tiers, Runway Gen-4.5 via `gen4_turbo`).2223> **Canonical model IDs** (Google IDs checked against ai.google.dev/gemini-api/docs/models on 2026-09-23):24>25> | Provider | Model IDs |26> |----------|-----------|27> | Anthropic | `claude-opus-5-5` (recommended, the default Opus since CC 2.1.280, same 2,576 px budget as Opus 5), `claude-opus-5`, `claude-opus-4-8`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-sonnet-5-5`, `claude-sonnet-4-6`, `claude-haiku-4-5-20251001`. `claude-fable-5` is Anthropic's **frontier tier above Opus** (GA 2026-07). Premium cost, never auto-pin it; the fable-spend-consent gate requires explicit user consent before any Fable spend |28> | OpenAI | `gpt-5.5` (current flagship) |29> | Google | `gemini-3.1-pro-preview` (flagship), `gemini-3.1-flash-lite` (cost) |30> | Veo | `veo-3.1-generate-preview` / `veo-3.1-lite-generate-preview` / `veo-3.1-fast-generate-preview` |31> | Kling | `kling-v3` (model_name field in Kling API) |32> | Runway | `gen4_turbo` (product label: Gen-4.5) |3334## Quick Reference3536| Category | Rules | Impact | When to Use |37|----------|-------|--------|-------------|38| [Vision: Image Analysis](#vision-image-analysis) | 1 | HIGH | Image captioning, VQA, multi-image comparison, object detection |39| [Vision: Document Understanding](#vision-document-understanding) | 1 | HIGH | OCR, chart/diagram analysis, PDF processing, table extraction |40| [Vision: Model Selection](#vision-model-selection) | 1 | MEDIUM | Choosing provider, cost optimization, image size limits |41| [Audio: Speech-to-Text](#audio-speech-to-text) | 1 | HIGH | Transcription, speaker diarization, long-form audio |42| [Audio: Text-to-Speech](#audio-text-to-speech) | 1 | MEDIUM | Voice synthesis, expressive TTS, multi-speaker dialogue |43| [Audio: Model Selection](#audio-model-selection) | 1 | MEDIUM | Real-time voice agents, provider comparison, pricing |44| [Video: Model Selection](#video-model-selection) | 1 | HIGH | Choosing video gen provider (Kling, Sora, Veo, Runway) |45| [Video: API Patterns](#video-api-patterns) | 1 | HIGH | Async task polling, SDK integration, webhook callbacks |46| [Video: Multi-Shot](#video-multi-shot) | 1 | HIGH | Storyboarding, character elements, scene consistency |4748**Total: 9 rules across 3 categories (Vision, Audio, Video Generation)**4950## Vision: Image Analysis5152Send images to multimodal LLMs for captioning, visual QA, and object detection. Always set `max_tokens` and resize images before encoding.5354| Rule | File | Key Pattern |55|------|------|-------------|56| Image Analysis | `rules/vision-image-analysis.md` | Base64 encoding, multi-image, bounding boxes |5758## Vision: Document Understanding5960Extract structured data from documents, charts, and PDFs using vision models.6162| Rule | File | Key Pattern |63|------|------|-------------|64| Document Vision | `rules/vision-document.md` | PDF page ranges, detail levels, OCR strategies |6566## Vision: Model Selection6768Choose the right vision provider based on accuracy, cost, and context window needs.6970| Rule | File | Key Pattern |71|------|------|-------------|72| Vision Models | `rules/vision-models.md` | Provider comparison, token costs, image limits |7374## Audio: Speech-to-Text7576Convert audio to text with speaker diarization, timestamps, and sentiment analysis.7778| Rule | File | Key Pattern |79|------|------|-------------|80| Speech-to-Text | `rules/audio-speech-to-text.md` | Gemini long-form, GPT-4o-Transcribe, AssemblyAI features |8182## Audio: Text-to-Speech8384Generate natural speech from text with voice selection and expressive cues.8586| Rule | File | Key Pattern |87|------|------|-------------|88| Text-to-Speech | `rules/audio-text-to-speech.md` | Gemini TTS, voice config, auditory cues |8990## Audio: Model Selection9192Select the right audio/voice provider for real-time, transcription, or TTS use cases.9394| Rule | File | Key Pattern |95|------|------|-------------|96| Audio Models | `rules/audio-models.md` | Real-time voice comparison, STT benchmarks, pricing |9798## Video: Model Selection99100Choose the right video generation provider based on use case, duration, and budget.101102| Rule | File | Key Pattern |103|------|------|-------------|104| Video Models | `rules/video-generation-models.md` | Kling vs Sora vs Veo vs Runway, pricing, capabilities |105106## Video: API Patterns107108Integrate video generation APIs with proper async polling, SDKs, and webhook callbacks.109110| Rule | File | Key Pattern |111|------|------|-------------|112| API Integration | `rules/video-generation-patterns.md` | Kling REST, fal.ai SDK, Vercel AI SDK, task polling |113114## Video: Multi-Shot115116Generate multi-scene videos with consistent characters using storyboarding and character elements.117118| Rule | File | Key Pattern |119|------|------|-------------|120| Multi-Shot | `rules/video-multi-shot.md` | Kling v3 character elements, 6-shot storyboards, identity binding |121122## Key Decisions123124| Decision | Recommendation |125|----------|----------------|126| High accuracy vision | `claude-opus-5-5` (default Opus since CC 2.1.280, 2,576 px vision budget, 3× what Opus 4.6 allotted; give it crop/analyze tools rather than more thinking, which is the cheaper lever on this model). (`claude-fable-5` is the frontier SOTA option, GA 2026-07 — premium cost, use only with explicit consent via the fable-spend-consent gate) |127| Long documents | `gemini-3.1-pro-preview` (1M+ context) |128| Cost-efficient vision | `gemini-3.1-flash-lite` (GA successor of the Flash-Lite preview, which was shut down 2026-05-25; shutdown scheduled 2027-05-07) |129| Video analysis | `gemini-3.1-pro-preview` (native video, supersedes 2.5 Pro) |130| Voice assistant | Grok Voice Agent on Grok 4.20 (fastest, <1s) |131| Emotional voice AI | Gemini Live API |132| Long audio transcription | `gemini-3.1-pro-preview` (9.5hr) |133| Speaker diarization | AssemblyAI or Gemini |134| Self-hosted STT | Whisper Large V3 |135| Character-consistent video | `kling-v3` (Character Elements 3.0) |136| Narrative video / storytelling | Sora 2 (best cause-and-effect coherence) |137| Cinematic B-roll | `veo-3.1-generate-preview` (camera control + polished motion) |138| Budget drafts | `veo-3.1-lite-generate-preview` (~$0.05/s, 720/1080p) |139| Mid-tier fast renders | `veo-3.1-fast-generate-preview` |140| Professional VFX | Runway `gen4_turbo` (Act-Two motion transfer) |141| High-volume social video | `kling-v3` Standard (~$0.20/video) |142| Open-source video gen | Wan 2.6 or LTX-2 |143| Lip-sync / avatar video | `kling-v3` (native lip-sync API) |144145## Example146147```python148import anthropic, base64149150client = anthropic.Anthropic()151with open("image.png", "rb") as f:152 b64 = base64.standard_b64encode(f.read()).decode("utf-8")153154response = client.messages.create(155 model="claude-opus-5-5",156 max_tokens=1024,157 messages=[{"role": "user", "content": [158 {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},159 {"type": "text", "text": "Describe this image"}160 ]}]161)162```163164## Common Mistakes1651661. Not setting `max_tokens` on vision requests (responses truncated)1672. Sending oversized images without resizing (>2048px)1683. Using `high` detail level for simple yes/no classification1694. Using STT+LLM+TTS pipeline instead of native speech-to-speech1705. Not leveraging barge-in support for natural voice conversations1716. Using deprecated models (GPT-4V, Whisper-1)1727. Ignoring rate limits on vision and audio endpoints1738. Calling video generation APIs synchronously (they're async — poll or use callbacks)1749. Generating separate clips without character elements (characters look different each time)17510. Using Sora for high-volume social content (expensive, slow — use Kling Standard instead)176177## Related Skills178179- `ork-extras:rag-retrieval` - Multimodal RAG with image + text retrieval180- `ork-extras:llm-integration` - General LLM function calling patterns181- `streaming-api-patterns` - WebSocket patterns for real-time audio182- `ork:demo-producer` - Terminal demo videos (VHS, asciinema) — not AI video gen