Multimodal LLM

Multimodal LLM Patterns

Integrate vision, audio, and video generation capabilities from leading multimodal models. Covers image analysis, document understanding, real-time voice agents, speech-to-text, text-to-speech, and AI video generation (Kling 3.0, Sora 2, Veo 3.1, Runway Gen-4.5).

Quick Reference

Total: 9 rules across 3 categories (Vision, Audio, Video Generation)

Vision: Image Analysis

Send images to multimodal LLMs for captioning, visual QA, and object detection. Always set max_tokens and resize images before encoding.

Vision: Document Understanding

Extract structured data from documents, charts, and PDFs using vision models.

Vision: Model Selection

Choose the right vision provider based on accuracy, cost, and context window needs.

Audio: Speech-to-Text

Convert audio to text with speaker diarization, timestamps, and sentiment analysis.

Audio: Text-to-Speech

Generate natural speech from text with voice selection and expressive cues.

Audio: Model Selection

Select the right audio/voice provider for real-time, transcription, or TTS use cases.

Video: Model Selection

Choose the right video generation provider based on use case, duration, and budget.

Video: API Patterns

Integrate video generation APIs with proper async polling, SDKs, and webhook callbacks.

Video: Multi-Shot

Generate multi-scene videos with consistent characters using storyboarding and character elements.

Key Decisions

Example

import anthropic, base64

client = anthropic.Anthropic()
with open("image.png", "rb") as f:
    b64 = base64.standard_b64encode(f.read()).decode("utf-8")

response = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=1024,
    messages=[{"role": "user", "content": [
        {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
        {"type": "text", "text": "Describe this image"}
    ]}]
)

Common Mistakes

1. Not setting max_tokens on vision requests (responses truncated)
2. Sending oversized images without resizing (>2048px)
3. Using high detail level for simple yes/no classification
4. Using STT+LLM+TTS pipeline instead of native speech-to-speech
5. Not leveraging barge-in support for natural voice conversations
6. Using deprecated models (GPT-4V, Whisper-1)
7. Ignoring rate limits on vision and audio endpoints
8. Calling video generation APIs synchronously (they're async — poll or use callbacks)
9. Generating separate clips without character elements (characters look different each time)
10. Using Sora for high-volume social content (expensive, slow — use Kling Standard instead)

Related Skills

• ork:rag-retrieval - Multimodal RAG with image + text retrieval
• ork:llm-integration - General LLM function calling patterns
• streaming-api-patterns - WebSocket patterns for real-time audio
• ork:demo-producer - Terminal demo videos (VHS, asciinema) — not AI video gen

Converted and distributed by TomeVault — claim your Tome and manage your conversions.