Comfy Director
Story first, pictures second. Multi-shot AI films fail in the edit, not the
render: beats that don't cause each other read as a montage wearing a story's
clothes, no matter how good each clip looks.
REQUIRED BACKGROUND: the comfy skill (CLI mechanics + the recipe path —
apply/capture/foreach — for reusable multi-stage graphs). On any failed job,
comfy-debug.
Order of operations
- Concept — one idea a stranger can retell.
- Screenplay — before touching the pipeline. Deliver it to the user first.
- Shot list — each scene mapped to a continuity strategy.
- Audio design — against the screenplay's emotional beats, not after picture.
- Produce → conform → QC.
Skipping 2 and jumping to renders is the #1 cause of killed cuts.
Screenplay rules
- One protagonist the camera can follow. Recognizable in every shot.
- Causality test: every cut answers "and BECAUSE of that…" — never
"and then, somewhere else…". If a beat can be reordered without breaking
anything, it isn't a story yet.
- Stranger test: someone with zero context must retell the plot in two
sentences ("X wanted A, B stopped them, they did C, so D").
- Beats carry timings that sum to the target runtime.
Concept rules
- The brand/product is felt, not name-checked: argue its philosophy;
in-jokes and community lore quoted on screen read as fan-service.
- A technique the client mentions ("style transfer", a model name) is a
spark, not a mandate — use it only if the idea needs it. Building the
film around a technique is the tell of a vendor, not a director.
- Weirdness must compound: surprising subject AND object AND setting.
One weird element in a normal frame is decoration.
Prompting shots
- Photoreal shots: live-action cinematographer / film-stock language ONLY.
One illustrator/anime/comics name in a photoreal prompt poisons it.
Illustrated references belong only in deliberately stylized shots.
- Motion prompts describe HOW the scene moves, not what's in it (see
comfy
skill video gotchas — t2v/i2v model traps, SaveVideo, fps wiring).
Continuity toolkit
Pick per shot, in rough order of strength:
| Strategy |
Use for |
| Character/location reference image carried into each shot's i2v graph |
protagonist identity |
| Same identity, new framings via reference-image edit (one hero → N angles) |
recognizable face across cuts |
| Keyframe of shot N seeds shot N+1 (image-referenced restyle or end-frame) |
contiguous action |
| Talking head fed the EXACT audio that plays (KlingAvatar), cut between angles of the same lip-synced head — never B-roll over the VO |
dialogue that must feel real |
| Repeated wardrobe/light/lens lines verbatim in every prompt |
cheap baseline glue |
If a shot can't hold continuity, rewrite the screenplay so it doesn't
need that shot — don't ship the break.
Audio
- Score = short designed cues joined at story pivots; one long generation
will not follow a dynamic arc (it inverts exactly where you need the peak).
- Place VO by math: stem duration vs beat window, before mixing.
- Duck score under VO; loudness-normalize the final mix (~-14 LUFS).
- Dialogue must lip-sync by construction. Drive the mouth from the exact
audio that plays (KlingAvatar
sound_file) and cut between angles of the same
lip-synced head — B-roll over a continuous VO reads as a montage, and
i2v-invented mouths drift. "No lip-sync / doesn't feel real" is the classic
rejection. (Generate audio in-graph — cloud LoadAudio can't see uploads; see
comfy audio gotchas.)
- Lock voice & casting by EAR before the expensive renders. Voice/accent is
subjective and unreadable from a waveform — render a few cheap candidate TTS
takes of one line, have the user pick, THEN bake the winner into the costly,
transient_auth-prone avatar/lip-sync shots. Re-rendering all of them on a
rejected voice is the avoidable burn. Same logic for the protagonist's look:
approve the hero still before generating every shot from it.
- Verify the mix by measurement, not faith. You can't hear the render —
ffmpeg volumedetect each section: a "dead air" ending or a buried line shows
up as a window near -91 dB. Confirm every beat is present at level before
declaring it done.
Production ops
Append EVERY submit — shots, keyframes, audio stems, resubmits — to
job_ids.txt (<job_name> <prompt_id>) at submit time. Append-only:
a resubmit gets a new line (shot_s3_resubmit <id>), never an overwrite,
so dead prompt_ids keep their shot identity.
Keep LOG.md at the project root: an append-only event log, written at
event time, not session end — one line per submit/resubmit (+ failure
code), per QC verdict (+ re-roll reason), and per decision that invalidates
earlier work: recasts, prompt changes, which reference image or stem is now
canonical. Frame-by-frame findings live in qc/QC_LOG.md; LOG.md indexes them.
17:36 ERROR shot_s3 transient_auth -> RESUBMIT 27dd45b4 (same workflow, no re-login)
17:44 QC shot_s2 FAIL background-crawl -> re-roll (same kf, new seed); recut beat if 2nd fails
17:50 DECISION recast protagonist -> ref_ray.png canonical; vo2, vo6 stale, re-gen
Recovery contract: a successor agent with zero memory must resume from
disk alone: LOG.md (what happened, why) → job_ids.txt (which jobs) →
comfy jobs status (their states). A decision not in LOG.md dies with the
session — completed-but-stale jobs (an old casting's VO stems) otherwise
look identical to good ones.
ffprobe every downloaded clip; normalize off-spec duration/dimensions
before conform.
QC one frame from every clip against the screenplay before cutting.
Exact runtime: trim/retime per-beat in ffmpeg, don't hope.
Common mistakes (each killed a real cut)
| Mistake |
Result |
| Renders before screenplay |
"Eight handsome beats that don't cause each other" |
| Community lore on screen |
"Too direct, fan-service" |
| Client's mentioned technique as the concept |
"I said it to spark ideas" |
| No continuity plan |
Protagonist changes face every cut |
| One 40s music gen |
Arc inverts at the climax |
| B-roll over a continuous VO |
"Feels like a montage / audio doesn't match" |
| Baked the voice into N shots before the user heard it |
Re-render everything when they reject it |
| Butt-joined talking-head clips |
Voice drifts out of sync (avatar pads video past audio) |
1---2name: comfy-director3description: Use when asked to make a narrative video — an ad, brand film, short, trailer, music video, or any multi-shot clip that tells a story (not a single shot or loop). Also use when a video was rejected for "no story", "doesn't flow", "feels like a montage", or characters that change between shots.4metadata:5 source: https://skillmd.com/skills/comfy-org/comfy-director6---78# Comfy Director910Story first, pictures second. Multi-shot AI films fail in the edit, not the11render: beats that don't cause each other read as a montage wearing a story's12clothes, no matter how good each clip looks.1314**REQUIRED BACKGROUND:** the `comfy` skill (CLI mechanics + the recipe path —15`apply`/`capture`/`foreach` — for reusable multi-stage graphs). On any failed job,16`comfy-debug`.1718## Order of operations19201. **Concept** — one idea a stranger can retell.212. **Screenplay** — before touching the pipeline. Deliver it to the user first.223. **Shot list** — each scene mapped to a continuity strategy.234. **Audio design** — against the screenplay's emotional beats, not after picture.245. **Produce** → **conform** → **QC**.2526Skipping 2 and jumping to renders is the #1 cause of killed cuts.2728## Screenplay rules2930- **One protagonist** the camera can follow. Recognizable in every shot.31- **Causality test:** every cut answers "and BECAUSE of that…" — never32 "and then, somewhere else…". If a beat can be reordered without breaking33 anything, it isn't a story yet.34- **Stranger test:** someone with zero context must retell the plot in two35 sentences ("X wanted A, B stopped them, they did C, so D").36- Beats carry timings that sum to the target runtime.3738## Concept rules3940- The brand/product is **felt, not name-checked**: argue its philosophy;41 in-jokes and community lore quoted on screen read as fan-service.42- A technique the client mentions ("style transfer", a model name) is a43 **spark, not a mandate** — use it only if the idea needs it. Building the44 film around a technique is the tell of a vendor, not a director.45- Weirdness must compound: surprising subject AND object AND setting.46 One weird element in a normal frame is decoration.4748## Prompting shots4950- **Photoreal shots:** live-action cinematographer / film-stock language ONLY.51 One illustrator/anime/comics name in a photoreal prompt poisons it.52 Illustrated references belong only in deliberately stylized shots.53- Motion prompts describe HOW the scene moves, not what's in it (see `comfy`54 skill video gotchas — t2v/i2v model traps, SaveVideo, fps wiring).5556## Continuity toolkit5758Pick per shot, in rough order of strength:5960| Strategy | Use for |61|---|---|62| Character/location reference image carried into each shot's i2v graph | protagonist identity |63| Same identity, new framings via reference-image edit (one hero → N angles) | recognizable face across cuts |64| Keyframe of shot N seeds shot N+1 (image-referenced restyle or end-frame) | contiguous action |65| Talking head fed the EXACT audio that plays (KlingAvatar), cut between angles of the *same* lip-synced head — never B-roll over the VO | dialogue that must feel real |66| Repeated wardrobe/light/lens lines verbatim in every prompt | cheap baseline glue |6768If a shot can't hold continuity, **rewrite the screenplay** so it doesn't69need that shot — don't ship the break.7071## Audio7273- Score = **short designed cues joined at story pivots**; one long generation74 will not follow a dynamic arc (it inverts exactly where you need the peak).75- Place VO by math: stem duration vs beat window, before mixing.76- Duck score under VO; loudness-normalize the final mix (~-14 LUFS).77- **Dialogue must lip-sync by construction.** Drive the mouth from the exact78 audio that plays (KlingAvatar `sound_file`) and cut between *angles of the same79 lip-synced head* — B-roll over a continuous VO reads as a montage, and80 i2v-invented mouths drift. "No lip-sync / doesn't feel real" is the classic81 rejection. (Generate audio in-graph — cloud `LoadAudio` can't see uploads; see82 `comfy` audio gotchas.)83- **Lock voice & casting by EAR before the expensive renders.** Voice/accent is84 subjective and unreadable from a waveform — render a few cheap candidate TTS85 takes of one line, have the user pick, THEN bake the winner into the costly,86 `transient_auth`-prone avatar/lip-sync shots. Re-rendering all of them on a87 rejected voice is the avoidable burn. Same logic for the protagonist's look:88 approve the hero still before generating every shot from it.89- **Verify the mix by measurement, not faith.** You can't hear the render —90 `ffmpeg volumedetect` each section: a "dead air" ending or a buried line shows91 up as a window near -91 dB. Confirm every beat is present at level before92 declaring it done.9394## Production ops9596- Append EVERY submit — shots, keyframes, audio stems, resubmits — to97 `job_ids.txt` (`<job_name> <prompt_id>`) **at submit time**. Append-only:98 a resubmit gets a new line (`shot_s3_resubmit <id>`), never an overwrite,99 so dead prompt_ids keep their shot identity.100- Keep `LOG.md` at the project root: an append-only event log, written **at101 event time, not session end** — one line per submit/resubmit (+ failure102 code), per QC verdict (+ re-roll reason), and per decision that invalidates103 earlier work: recasts, prompt changes, which reference image or stem is now104 canonical. Frame-by-frame findings live in `qc/QC_LOG.md`; LOG.md indexes them.105106 ```107 17:36 ERROR shot_s3 transient_auth -> RESUBMIT 27dd45b4 (same workflow, no re-login)108 17:44 QC shot_s2 FAIL background-crawl -> re-roll (same kf, new seed); recut beat if 2nd fails109 17:50 DECISION recast protagonist -> ref_ray.png canonical; vo2, vo6 stale, re-gen110 ```111- **Recovery contract:** a successor agent with zero memory must resume from112 disk alone: `LOG.md` (what happened, why) → `job_ids.txt` (which jobs) →113 `comfy jobs status` (their states). A decision not in LOG.md dies with the114 session — completed-but-stale jobs (an old casting's VO stems) otherwise115 look identical to good ones.116- ffprobe every downloaded clip; normalize off-spec duration/dimensions117 before conform.118- QC one frame from every clip against the screenplay before cutting.119- Exact runtime: trim/retime per-beat in ffmpeg, don't hope.120121## Common mistakes (each killed a real cut)122123| Mistake | Result |124|---|---|125| Renders before screenplay | "Eight handsome beats that don't cause each other" |126| Community lore on screen | "Too direct, fan-service" |127| Client's mentioned technique as the concept | "I said it to spark ideas" |128| No continuity plan | Protagonist changes face every cut |129| One 40s music gen | Arc inverts at the climax |130| B-roll over a continuous VO | "Feels like a montage / audio doesn't match" |131| Baked the voice into N shots before the user heard it | Re-render everything when they reject it |132| Butt-joined talking-head clips | Voice drifts out of sync (avatar pads video past audio) |