Storyboard Prompt Engineering

Storyboard Prompt Engineering

Prompt engineering toolkit for 4-pass AI video storyboard pipelines (LangGraph). Covers modular rule composition, filter bypass, and retry consistency.

When to Use

• Adding/modifying prompt rules for storyboard generation
• Implementing or adjusting dodge layers for video API content filters (Kling/Jimeng/Seedance)
• Fixing retry consistency issues (shots being rewritten unnecessarily)
• Creating custom PromptSkill modules for new genres/styles

Architecture

Pipeline (LangGraph StateGraph)
├── Pass 1: Scene     ← core + style + dodge
├── Pass 2: Character ← core + physics + dodge
├── Pass 3: Shot      ← core + dialogue + physics + dodge + continuity*
├── Pass 4: Verify    ← core + dialogue + dodge
└── Retry (score<10)  ← prepareRetry → continuity activates

Post-process: sanitizer.ts applies regex replacement + dodge layer injection on final output (covers desc, act, motive in seq).

Output Schema (StoryboardResponse.seq)

Each shot in seq contains:

act/fx/motive are optional fields populated by the 4-Pass Pipeline's aggregate layer. The single-pass LangChain path only produces {id, desc}.

PromptSkill Quick Reference

interface PromptSkill {
  id: string
  rules: string | ((state: PipelineStateSlice) => string)
  appliesTo: PassType[]  // 'scene' | 'character' | 'shot' | 'verify'
  priority: number       // lower = earlier in prompt
}

Priority Ranges

• 0-9: Core constraints (always first)
• 10-19: Domain/genre rules (style, physics, dialogue, custom genres)
• 20-29: Safety/compliance (dodge)
• 30+: Context-dependent (continuity, injected at runtime)

Adding a Custom Skill

import { BUILTIN_SKILLS, type PromptSkill } from './prompt-skills'

const horrorSkill: PromptSkill = {
  id: 'horror',
  rules: `Horror Rules:
- Shadow depth > 85%, single cold light source
- Sound: low-freq drone 30-50Hz + sudden silence gaps
- fg must have occluder (door frame/fingers/bars) for voyeuristic framing`,
  appliesTo: ['scene', 'shot'],
  priority: 15
}

const service = new StoryboardPipelineService(config, [...BUILTIN_SKILLS, horrorSkill])

Dodge Layer (Dual Defense)

Two independent layers — both required:

Layer 1 (Prompt-level): D1-D8 rules in system prompt guide LLM to generate safe text. See references/dodge-patterns.md.

Layer 2 (Post-process): sanitizer.ts runs regex replacement + dodge modifier injection on final output.

CRITICAL CONSTRAINTS:

• Do NOT merge the two layers — they serve different purposes (generation guidance vs output cleanup)
• Do NOT add platform-specific rules without validated evidence of differing filter behavior
• injectDodgeLayer appends to desc with | separator — never insert mid-string

Continuity Lock

Activates ONLY during retry (when state.retryFeedback is non-empty).

Token Budget

previousShots stores { id, desc } only — NOT full ShotData.

Do NOT expand PreviousShot to full ShotData — token budget would jump to ~3000+ and crowd out generation context. The desc field contains the 5-part shot specification which is sufficient for continuity reference.

How It Works

1. prepareRetry saves shots.map(s => ({id, desc})) as previousShots
2. buildContinuityLock(state) generates lock rules with character anchors + reference frames
3. Lock rule injected into shot pass system prompt at priority 30 (after all other rules)
4. Rule instructs LLM: unmentioned shots → preserve verbatim

See references/continuity-lock.md for implementation details.

Common Mistakes