Video Tools

Video Tools

Use this skill when the user asks to create, generate, produce, edit, extend, or modify videos and animations.

Loading this skill activates the generate_video and edit_video tools.

IMPORTANT: After loading this skill, you MUST call either generate_video or edit_video tool to perform video operations. Do NOT call video_tools directly - it is NOT a tool name.

Important: Video generation is asynchronous and takes 1-5 minutes. The tools return a task ID immediately; the user will be notified when the result is ready.

Tool: generate_video

Generate a video from text, images, videos, audio references, or any combination using an AI video generation model.

When to Use

• User asks to create, generate, or produce a video, animation, or motion content.
• User wants to generate a video using reference images, videos, or audio as creative inputs.

Mode Selection Guide

Choose the correct video_mode based on the user's input:

CRITICAL: When the user provides 2 or more images as references (e.g. "use image A and image B to generate a video"), you MUST use video_mode="reference_images" with the reference_images array. Do NOT use image_to_video — that mode only accepts a single first-frame image and will ignore all other images.

Audio Generation (Dialogue, Narration, Sound Effects)

IMPORTANT: Many video models (e.g. Gemini Veo 3.0/3.1, Seedance 2.0) are fully multimodal — they can generate videos with audio including:

• Dialogue — characters speaking lines
• Narration/Voiceover — off-screen narrator voice
• Sound effects — environmental sounds, footsteps, etc.
• Ambient audio — background atmosphere

You do NOT need reference audio to generate videos with dialogue. Simply describe the dialogue, narration, or sound effects in the prompt, and the model will generate them automatically.

Example prompts with audio:

# Dialogue example
prompt="A woman walks into the coffee shop and says 'Good morning!' to the barista with a warm smile, ambient coffee shop sounds, cinematic lighting"

# Narration example  
prompt="A documentary-style shot of a cheetah running across the savanna, narrator voice describing 'The cheetah can reach speeds of 70 miles per hour', dramatic music builds"

# Sound effects example
prompt="A car speeds through a rainy city street at night, tires splashing through puddles, engine roaring, rain pattering on the windshield"

When to use reference_audios:

• To clone a specific voice style or tone from an existing audio sample
• To match music genre/mood from a reference track
• The model will inherit voice characteristics or musical style from the reference

Parameters

Examples

Text to video:

generate_video(
  prompt="A golden retriever running through a sunlit meadow with wildflowers, cinematic slow motion",
  aspect_ratio="16:9",
  duration=6,
  quality="720p"
)

Image to video (first frame):

generate_video(
  prompt="The character in the image starts walking forward with gentle wind blowing",
  video_mode="image_to_video",
  image_url="/api/media/character.jpg",
  duration=6
)

First + last frame video:

generate_video(
  prompt="Camera slowly zooms in from the first frame scene to the last frame close-up",
  video_mode="image_to_video",
  image_url="/api/media/scene_wide.jpg",
  last_frame_image="/api/media/scene_closeup.jpg",
  duration=8
)

Multimodal reference (2 images — most common scenario):

generate_video(
  prompt="图片1中的男生牵着图片2中的女生的手，在赛博朋克城市街道上奔跑，霓虹灯闪烁，雨水反射灯光",
  video_mode="reference_images",
  reference_images=["/api/media/boy.jpg", "/api/media/girl.jpg"],
  duration=8,
  aspect_ratio="16:9"
)

Multimodal reference (images + video + audio):

generate_video(
  prompt="使用视频1的第一视角构图，使用音频1作为背景音乐。图片1中的角色走进图片2的场景",
  video_mode="reference_images",
  reference_images=["/api/media/character.jpg", "/api/media/scene.jpg"],
  reference_videos=["/api/media/camera_movement.mp4"],
  reference_audios=["https://example.com/bgm.mp3"],
  duration=10,
  aspect_ratio="16:9"
)

Tool: edit_video

Edit, modify, or extend an existing video using AI. Supports replacing objects, changing styles, extending footage, and concatenating multiple video clips.

When to Use

• User asks to modify, stylize, replace objects, or add effects to an existing video (edit mode).
• User asks to extend a video by generating additional frames (extend mode).
• User asks to concatenate multiple video clips into one continuous video (extend mode with additional_videos).

Parameters

Examples

Edit video (replace object):

edit_video(
  video_url="/api/media/product_ad.mp4",
  prompt="将视频1中的香水替换成图片1中的面霜，运镜不变",
  mode="edit",
  reference_images=["https://example.com/cream.jpg"],
  duration=5
)

Extend video:

edit_video(
  video_url="/api/media/scene.mp4",
  prompt="The camera slowly pans to reveal a mountain landscape in the background",
  mode="extend",
  duration=6
)

Concatenate multiple videos:

edit_video(
  video_url="/api/media/clip1.mp4",
  prompt="视频1中的窗户打开，进入室内，接视频2，之后镜头进入画内，接视频3",
  mode="extend",
  additional_videos=["/api/media/clip2.mp4", "/api/media/clip3.mp4"],
  duration=8
)

Tips

• Always write video prompts in English for best quality, except when using Chinese-specific features (refer to media as 图片1, 视频1, 音频1).
• Video generation is async — inform the user it will take 1-5 minutes.
• For image_to_video, use a local media path (e.g. /api/media/xxx.jpg) from canvas nodes or previous generations.
• For multimodal reference, you can combine images, videos, and audio freely. The model will inherit visual style, camera movement, and audio characteristics from the references.
• First frame/last frame mode and multimodal reference mode are mutually exclusive — do not mix them.
• Use return_last_frame=true to chain consecutive video generations (last frame becomes next first frame).
• Duration of -1 lets the model auto-select the optimal duration.
• Not all models support all modes and parameters. The available options are automatically tailored to the configured model.

Error Handling

When the video generation API returns an error, the error message will be passed back to you. Handle these common errors:

CRITICAL: When you receive a content safety rejection error, you MUST:

1. Stop all retry attempts immediately
2. Explain the specific reason to the user clearly
3. Suggest alternatives (e.g. use AI-generated character images instead of real photos)
4. Wait for the user to provide new input before trying again

Canvas Node → Multimodal Reference Workflow

Canvas nodes are identified by UUIDs. To use canvas media as multimodal references:

Step 1: Call list_canvas_nodes to discover available image/video nodes:

list_canvas_nodes(node_type="image")  → returns [{id: "uuid-a", name: "角色A"}, ...]
list_canvas_nodes(node_type="video")  → returns [{id: "uuid-x", name: "片段1"}, ...]

Step 2: Call get_canvas_node(node_id="uuid-a") to get the media URL:

• Image nodes → data.imageUrl (e.g. /api/media/character.jpg)
• Video nodes → data.videoUrl (e.g. /api/media/clip1.mp4)

Step 3: Pass URLs to generate_video/edit_video in the desired order. The array order determines the number in the prompt:

Example: User says "用角色A和场景B生成视频，参考片段C的运镜"

# 1. Get media URLs from canvas nodes
角色A_url = get_canvas_node("uuid-a").data.imageUrl  → /api/media/characterA.jpg
场景B_url = get_canvas_node("uuid-b").data.imageUrl  → /api/media/sceneB.jpg
片段C_url = get_canvas_node("uuid-c").data.videoUrl  → /api/media/clipC.mp4

# 2. Call generate_video with ordered arrays
generate_video(
  prompt="图片1中的角色走进图片2的场景中，使用视频1的运镜方式",
  video_mode="reference_images",
  reference_images=[角色A_url, 场景B_url],   # index 0=图片1, index 1=图片2
  reference_videos=[片段C_url],              # index 0=视频1
  duration=8
)