Video Gen

AI Video Generation

Generate videos from text prompts or images using diffusion-based models. This is the umbrella skill for all AI video generation providers.

Companion skills: video-editor · remotion · faceless-youtube · auto-clipper · b-roll-finder

When to Use

Trigger phrases:

• "generate a video" · "make a video from text" · "text to video" · "image to video"
• "AI video" · "animate this image" · "create a video clip" · "video generation"
• "seedance" · "即梦" · "runway" · "kling video" · "sora" · "grok video" · "pika"
• "extend this video" · "continue this video" · "video from prompt"

Use cases:

• Generate short video clips (4-60s) from text descriptions
• Animate still images into dynamic video
• Extend existing videos with AI-generated continuation
• Create product demos, ads, and social media clips
• Generate B-roll footage for longer productions
• Multi-modal generation (image + video + audio references)

When NOT to use:

• For programmatic/data-driven video (use remotion)
• For post-production editing of existing footage (use video-editor)
• For building YouTube channels at scale (use faceless-youtube)
• When you need frame-precise control (use remotion)

Overview

AI video generation creates video content from text prompts or images using diffusion-based models. Different providers excel at different tasks — this skill routes to the right one.

Model Selection Guide

When to pick which

• Product ad / e-commerce → Seedance 2.0 (360° spins, multi-angle, Chinese prompts)
• Cinematic scene → Kling 3.0 or Sora (highest quality)
• Quick social clip → Grok Imagine (fast, with audio)
• Image animation → Runway Gen-3 or Wan 2.7
• Video extension → Veo 3-1 or Seedance 2.0
• Lip-sync / voiceover → Wan 2.7 or Seedance 2.0

Prompt Engineering

Universal Formula

[Subject] + [Action] + [Camera] + [Style] + [Mood] + [Audio]

Camera Vocabulary

Style Vocabulary

Good vs Bad Prompts

❌ "A car driving"
✅ "A red sports car driving on a coastal highway at sunset, aerial tracking shot, cinematic, 4K, warm golden light"

❌ "A person talking"
✅ "Close-up of a woman speaking passionately, shallow depth of field, soft studio lighting, professional interview style"

Negative Prompting (Seedance, Runway)

no watermarks, no subtitles, no text overlays, no blur, no artifacts, no distortion

Provider-Specific APIs

Runway Gen-3

import requests, time

response = requests.post(
    "https://api.dev.runwayml.com/v1/image_to_video",
    headers={
        "Authorization": f"Bearer {RUNWAY_API_KEY}",
        "Content-Type": "application/json",
    },
    json={
        "model": "gen3a_turbo",
        "promptImage": "https://example.com/frame.png",
        "promptText": "Camera slowly pans right, subject walks forward",
        "duration": 5,
        "ratio": "16:9",
    },
)
task_id = response.json()["id"]

# Poll for completion
while True:
    status = requests.get(
        f"https://api.dev.runwayml.com/v1/tasks/{task_id}",
        headers={"Authorization": f"Bearer {RUNWAY_API_KEY}"},
    ).json()
    if status["status"] == "SUCCEEDED":
        video_url = status["output"][0]
        break
    time.sleep(5)

Kling

response = requests.post(
    "https://api.klingai.com/v1/videos/text2video",
    headers={"Authorization": f"Bearer {KLING_API_KEY}"},
    json={
        "prompt": "A golden retriever running through autumn leaves, slow motion, cinematic",
        "model_version": "kling-v1",
        "duration": "5",
        "aspect_ratio": "16:9",
    },
)

Seedance 2.0 (即梦)

Platform: jimeng.jianying.com · Video Length: 4-15s per generation

Multi-modal references:

• Images: @图片1 ~ @图片9 (character consistency, scene, product angles)
• Videos: @视频1 ~ @视频3 (camera motion, action patterns, pacing)
• Audio: @音频1 ~ @音频3 (music, voice tone, SFX)

10 core capabilities: text gen, consistency control, camera replication, VFX replication, story completion, video extension, sound control, one-take long shot, video editing, music beat sync

E-commerce template:

@图片1中的[产品]，360度高速旋转2圈后，突然停住蓄力分裂成了3个部分进行展示。
3D渲染产品展示特效，动感产品特效展示

Timestamp storyboarding (13-15s):

0-3s: [opening scene]
4-8s: [main action]
9-12s: [climax]
13-15s: [closing shot]

Technical params: 16:9 / 9:16 / 1:1 · 24fps (cinematic) / 30fps (smooth) · color grading: cinematic, cyberpunk, moody, fresh · lens: wide-angle, telephoto, macro, fisheye

Grok Imagine (Aurora AI)

Access: Grok mobile app (iOS/Android) · Requires: Super Grok subscription Modes: Normal (balanced) · Fun (creative) · Custom (fine control)

"A majestic eagle soaring above snow-capped mountains at dawn,
tracking shot, cinematic epic style, wind sounds and ambient nature audio"

RunComfy CLI (Wan 2.7, Veo 3-1, HappyHorse)

npm i -g @runcomfy/cli
runcomfy login

# Text-to-video
runcomfy run wan-ai/wan-2-7/text-to-video --input '{"prompt":"..."}'

# Image-to-video with lip-sync
runcomfy run wan-ai/wan-2-7/image-to-video --input '{"image_url":"...","audio_url":"..."}'

# Video extension
runcomfy run google-deepmind/veo-3-1/extend-video --input '{"video_url":"..."}'

Common Patterns

Image-to-Video Pipeline

async def animate_image(image_path: str, prompt: str):
    with open(image_path, "rb") as f:
        upload = requests.post(
            "https://api.runwayml.com/v1/uploads",
            files={"file": f},
            headers={"Authorization": f"Bearer {RUNWAY_API_KEY}"},
        )
    image_url = upload.json()["url"]

    response = requests.post(
        "https://api.dev.runwayml.com/v1/image_to_video",
        json={"promptImage": image_url, "promptText": prompt, "duration": 5},
        headers={"Authorization": f"Bearer {RUNWAY_API_KEY}"},
    )
    return response.json()["id"]

# Batch animate product images
images = ["product1.png", "product2.png", "product3.png"]
tasks = [animate_image(img, "Product rotates slowly, studio lighting") for img in images]

Multi-Segment Video (>15s)

For videos longer than one generation's cap, split into segments with explicit continuity points:

Segment 1 (0-10s): Wide shot of city skyline at dusk, camera slowly zooms in
Segment 2 (10-20s): Camera continues zooming into a specific building window
Segment 3 (20-30s): Interior shot, person working at desk, soft lighting

Video Upscaling (Post-Processing)

from realesrgan import RealESRGAN
import cv2

model = RealESRGAN(scale=4)
model.load_weights("RealESRGAN_x4.pth")

cap = cv2.VideoCapture("input.mp4")
while True:
    ret, frame = cap.read()
    if not ret: break
    upscaled = model.predict(frame)
    cv2.imwrite(f"frames/{frame_num:06d}.png", upscaled)

Anti-Rationalization Table

Process

1. Select model — Pick the right provider for the task (see Model Selection Guide)
2. Craft prompt — Follow [Subject + Action + Camera + Style + Mood] formula
3. Generate variations — Run 3-5 generations, pick the best
4. Post-process — Upscale, color grade, add audio if needed
5. Export — Format for target platform

Verification

• Model selected based on task requirements, not habit
• Prompt follows [Subject + Action + Camera + Style + Mood] formula
• Negative prompts included where supported
• Multiple variations generated and best selected
• Output resolution and duration match requirements
• Post-processing applied if needed (upscale, color grade, audio)