MiniMax AI ComfyUI Drama
A 7-stage pipeline for producing a short drama video from a storyboard. The Plugin ships two preset ComfyUI workflows (scenarios 5 and 6) and describes the rest of the pipeline. The Plugin does not distribute any character LoRA, voice sample, or video model — every asset is user-supplied.
The 2 trigger scenarios (Part of this Skill)
These are scenarios 5 and 6 from comfyui-studio's index.
| # | Trigger | User says | Preset workflow | What it produces |
|---|---|---|---|---|
| 5 | 首帧 (first frame) | "生成这部短剧第 X 镜的首帧" / "first frame of shot 3" | workflows/drama-first-frame.json |
One 16:9 still image per shot, with two character LoRA slots |
| 6 | 出片 (image to video) | "把这个首帧做成 4 秒视频" / "turn this first frame into a clip" | workflows/drama-image-to-video.json |
One MP4 clip per shot, ~4 seconds, 24 FPS, distilled LTX-2.3 |
The Plugin provides templates and preset workflows for both stages. The user runs stages 1, 2, 3, 6, and 7 of the surrounding pipeline with their own tools (Excel, TTS pipeline, FFmpeg). This boundary is intentional: the Plugin does not assume a particular TTS vendor or spreadsheet format.
The 7 stages (the wider pipeline)
| Stage | Output | Tool | Notes |
|---|---|---|---|
| 1. Storyboard | desktop/<name>_分镜.xlsx |
User's spreadsheet editor or any storyboard tool | 6-column format: 序号 / 时长 / 画面 / 图片 prompt / 视频 prompt / 台词 |
| 2. TTS dubbing | <name>_wav/00X_<role>.wav |
User's TTS pipeline (Edge TTS, IndexTTS2, Qwen3-VoiceDesign, etc.) | TTS comes BEFORE images — the actual measured duration is the only truth for shot length |
| 3. Script refinement | Updated Excel with measured shot lengths | User's scripting tool | Re-cut shots whose actual TTS length differs from the script estimate by > 0.5s |
| 4. First-frame image | drama_frame_00001_.png per shot |
Scenario 5: workflows/drama-first-frame.json |
One image per shot, two-character LoRA slots |
| 5. Image-to-video clip | drama_clip_00001_.mp4 per shot |
Scenario 6: workflows/drama-image-to-video.json |
One video per shot, 8 GB VRAM friendly |
| 6. Subtitle burn | Subtitled video per shot | FFmpeg drawtext filter |
Source Han Sans CN, 28pt, white + black 2px stroke |
| 7. Audio/video assembly | Final drama MP4 | FFmpeg -c:v copy -c:a aac + concat demuxer |
One output MP4 per drama |
Scenario 5 — 首帧 (first frame) — drama-first-frame.json
A first-frame image generator with two LoRA slots (one per character in a two-character
drama). For single-character dramas, set the second slot's LoRA file to a placeholder LoRA
with strength = 0; for two-character dramas, fill both slots.
Pipeline: Checkpoint → LoraLoader (character A) → LoraLoader (character B) → KSampler.
Models
| Field | Value (as written in the JSON) | Where it lives on disk |
|---|---|---|
CheckpointLoaderSimple.ckpt_name |
Z-Image-Base-8steps-豹豹喵呜の白玉v2White_Marble-AIO_v2-bf16.safetensors |
models/checkpoints/ |
LoraLoader[0].lora_name (character A) |
character_a_lora.safetensors (placeholder — replace) |
models/loras/ |
LoraLoader[1].lora_name (character B) |
character_b_lora.safetensors (placeholder — replace) |
models/loras/ |
| CLIP / VAE | bundled with the CheckpointLoader | — |
Knobs
| Knob | Default | What it does |
|---|---|---|
LoraLoader[0].strength_model (character A) |
0.8 | Identity weight for character A |
LoraLoader[1].strength_model (character B) |
0.8 | Identity weight for character B |
EmptyLatentImage.width / height |
1280 / 720 | 16:9 landscape. Change for vertical (9:16) or square (1:1) |
KSampler.steps |
8 | Distilled-model default; raise to 20 for a non-distilled base |
KSampler.cfg |
1.5 | Distilled-model default; raise to 6 for a non-distilled base |
Submit
- Place the two character LoRAs under
<ComfyUI>/models/loras/. Edit the twoLoraLoader.lora_namefields in the JSON to point at your filenames if they differ from the placeholders. - Pass the image prompt with
--prompt. For a two-character drama, lead with the trigger words for both characters so the LoRAs can light up. - Submit via the Python script or the MCP server.
Critical prompt rule (first-frame)
The image prompt describes the first frame, not the shot's action sequence. A shot where character A hands a sword to character B should prompt the moment the hands are meeting, not the full handoff motion. The video prompt (scenario 6) is responsible for the motion.
Scenario 6 — 出片 (image to video) — drama-image-to-video.json
An image-to-video generator for a single shot. Takes a first-frame image (from scenario 5) and the shot's video prompt (Excel column "视频 prompt"). Outputs a single MP4 per shot.
Pipeline: CheckpointLoader + DualCLIPLoader + VAELoader + LoraLoader (optional motion) +
LoadImage (first frame) + KSampler + VHS_VideoCombine.
Models
| Field | Value (as written in the JSON) | Where it lives on disk |
|---|---|---|
CheckpointLoaderSimple.ckpt_name |
10Eros_v1-fp8mixed_learned.safetensors (LTX-2.3 22B finetune) |
models/checkpoints/ |
DualCLIPLoader.clip_name1 |
ltx-2.3_text_projection_bf16.safetensors |
models/clip/ |
DualCLIPLoader.clip_name2 |
gemma_3_12B_it_fp8_scaled.safetensors |
models/clip/ |
VAELoader.vae_name |
LTX23_video_vae_bf16.safetensors |
models/vae/ |
LoraLoader.lora_name (motion) |
any_motion_lora.safetensors (placeholder — replace, or set strength_model = 0 to disable) |
models/loras/ |
LoadImage.image |
first_frame.png (rename per shot, or pass --filename to drive the workflow from the CLI) |
input/ |
Knobs
| Knob | Default | What it does |
|---|---|---|
LoadImage.image |
"first_frame.png" | The first-frame image to animate. Use predictable filenames per shot and pass them via --filename on the CLI. |
VHS_VideoCombine.frame_rate |
24 | Output FPS. Pair with RIFE interpolation to reach 60 |
VHS_VideoCombine.format |
"video/h264-mp4" | Output container/codec |
KSampler.steps |
8 | Distilled LTX default; raise for non-distilled |
KSampler.cfg |
1.0 | Distilled LTX default |
Submit
- Place the first-frame image in the ComfyUI input folder with a predictable name
(
shot_001.png,shot_002.png, ...). - Pass the motion prompt (Excel "视频 prompt" column value) with
--prompt. The video prompt describes motion only — see the critical rule below. - Submit via the Python script or the MCP server. The script will poll until done and
download the resulting MP4 from
outputs/.
Critical prompt rule (motion-only)
The video prompt describes motion only — camera move, character animation, environment
dynamics, expression changes. It does not describe the visible image (the model can see
the first frame and already knows what is in it). The shot's dialogue goes in a separate
dialogue field, not in the prompt.
How to run a full drama (loop scenarios 5 and 6)
For each shot in the storyboard:
- Scenario 5: submit
drama-first-frame.jsonwith the image prompt → producesdrama_frame_00001_.png. - Scenario 6: rename / move the first-frame to
shot_001.pngin the ComfyUI input folder, then submitdrama-image-to-video.jsonwith the motion prompt → producesdrama_clip_00001_.mp4. - Stage 6 (out of scope of this Plugin): burn subtitles with FFmpeg
drawtext. - Stage 7 (out of scope of this Plugin): concat the clips and mux the TTS audio with FFmpeg.
Each shot is one ComfyUI submission per scenario (because each shot has a different first-frame image and a different prompt). The Plugin does not implement a batch driver; if you need one, use a wrapper loop in your own tool.
A note on long jobs and VRAM
A distilled LTX-class video model running at 1280×720 with 24 FPS for a 4-second clip finishes
in roughly 2–5 minutes on an 8 GB GPU. A non-distilled 22B video model on the same shot can
take 30–60 minutes. Long jobs (over 30 minutes) should be watched with nvidia-smi; the Plugin
does not ship a VRAM watchdog. If your GPU is lost mid-run, the standard recovery is to
interrupt the queue and restart ComfyUI.
What this Skill does not do
- It does not run LoRA training. Training is its own project; the Skill only consumes a LoRA the user already has.
- It does not bundle or distribute any LoRA, model, voice sample, face embedding, or any other identity asset. Every character and voice is user-supplied.
- It does not run TTS, edit Excel, burn subtitles, or assemble audio/video. Those are user pipeline steps that the Plugin deliberately does not assume.
- It does not ship a VRAM watchdog or auto-retry on failure. Long jobs need a human watching.
Requirements
- ComfyUI running locally (see
comfyui-workflow/SKILL.mdfor the transport). - A first-frame checkpoint and (optionally) two character LoRAs the user trained, dropped under
models/checkpoints/andmodels/loras/. The workflow JSONs reference the LoRAs ascharacter_a_lora.safetensorsandcharacter_b_lora.safetensors— replace with your filenames. - A distilled LTX-Video checkpoint and its CLIP / VAE files. The motion LoRA is referenced as
any_motion_lora.safetensors— replace or set its strength to 0 to disable. - A TTS pipeline (any of the user's choice) for stage 2.
- FFmpeg for stages 6 and 7.
License
Apache-2.0. See LICENSE.