Create Movie

create-movie

Orchestrated movie creation for Horus persona. Creates mockumentaries, short films, music videos, and educational content through a phased workflow.

Philosophy

"AI isn't the artist, it's the amplifier" - Nobody & The Computer

Horus uses AI to turn imagination into audiovisual reality. He doesn't just use pre-built tools - he writes code to create his own tools.

Phases

HARDWARE CHECK → RESEARCH → SCRIPT → CASTING → EXPERT REVIEW → GENERATE → ASSEMBLE → LEARN

Note: BUILD TOOLS is optional and only triggered when custom effects are needed.

Phase 0: Hardware Detection (Automatic)

Before any generation, the orchestrator automatically detects hardware via /ops-workstation:

# Automatic hardware check on startup
./run.sh create "prompt"
# → Calls /ops-workstation gpu to detect VRAM
# → Calls /ops-workstation memory to detect RAM
# → Auto-selects optimal model variant

Auto-Selection Logic:

RAM-Based Optimizations:

Override Auto-Detection:

# Force specific model variant
./run.sh create "prompt" --model ltx2-fp4
./run.sh create "prompt" --model ltx2-distilled
./run.sh create "prompt" --runpod  # Force cloud generation

Phase 1: Research (Library-First)

1. Check Horus's Library First:
   • horus-filmmaking scope (past techniques, learnings)
   • horus_lore scope (YouTube transcripts, film analysis)
   • Ingested movies with emotion tags
   • Episodic archive (past filmmaking sessions)
2. Search for New Resources:
   • /ingest-movie search for films to watch
   • /ingest-youtube search for tutorials
3. Deep Web Research:
   • /dogpile for comprehensive multi-source search
   • /surf for specific tutorials/references

Phase 2: Script (via /create-story)

• Integrates with /create-story skill for screenplay generation
• Uses Chutes models (chimera, qwen, deepseek-r1) for creative writing
• Parses INT./EXT. headings, dialogue, action, audio cues
• Outputs structured scene breakdown with visual descriptions

Format Options:

• screenplay (default) - Standard INT./EXT. scene headings
• mockumentary - Interview segments with talking heads + B-roll
• reconstruction - Historical recreation with narrator framing

Phase 2.5: Casting (via /create-cast)

• Multi-round collaborative character casting
• Extracts characters from screenplay via script analysis
• Optional reference actor discovery via /discover-talent (TMDB)
• Generates identity packs: front, 3/4, full-body reference images
• Voice casting from existing TTS models or queues training
• Bridge attributes extracted via Federated Taxonomy

Output:

characters/
├── casting_session.json
├── SARAH/
│   ├── character_bible.yaml
│   └── identity_pack/
│       ├── front.png
│       ├── three_quarter.png
│       └── full_body.png
└── voice_assignments.yaml

Phase 2.7: Expert Review (NON-NEGOTIABLE)

When a creative team has multiple personas, this phase enforces a feedback loop before generation:

1. DIRECTOR creates vision (shot list, style notes)
2. TECHNICAL EXPERT reviews for AI video execution
   - e.g., Dan Kieft for Kling, video model specialist for Veo
3. EXPERT sends formal notes to DIRECTOR
   - Technical limitations, proposed changes, questions
4. DIRECTOR approves, revises, or escalates
5. BOTH sign off before generation proceeds

Required Documents:

Expert Personas (queryable via /memory):

• Dan Kieft (scope: dan-kieft) - Kling AI, multi-shot prompting, character consistency
• Video model specialists - Add via /ingest-youtube + /memory learn

Workflow:

# 1. Create initial instructions
# (output: KLING_INSTRUCTIONS_V1.md)

# 2. Query expert persona
./run.sh recall --q "multi-shot prompting" --scope dan-kieft

# 3. Generate expert review
# (output: DAN_KIEFT_REVIEW.md)

# 4. Send to director for approval
# (output: DAN_TO_WILSON_FEEDBACK.md)

# 5. Director approves
# (output: WILSON_APPROVAL.md)

# 6. Final approved instructions
# (output: KLING_INSTRUCTIONS_V2_APPROVED.md)

# 7. ONLY NOW proceed to generation

Skip Conditions:

• Single-persona projects (no creative team)
• --skip-expert-review flag (use with caution)

Phase 3: Build Tools (Optional)

• Write code in Docker-isolated sandbox
• Create custom tools for specific effects
• Iterate on approaches

Phase 4: Generate

• Use ComfyUI, Stable Diffusion for images
• Use auto-selected video model based on hardware (LTX-2 FP8/FP4/Distilled)
• Use Whisper, IndexTTS2 for audio
• If hardware insufficient, automatically suggests /ops-runpod

Phase 5: Assemble

• Combine assets with FFmpeg
• Output MP4 video or interactive HTML

Phase 6: Learn

• Store successful techniques in /memory
• Remember what worked for future movies

Quick Start

cd .pi/skills/create-movie

# Full orchestrated workflow (recommended)
./run.sh create "A 30-second film about discovering colors"

# With options
./run.sh create "film noir detective" \
    --duration 60 \
    --style "high contrast, shadows, venetian blinds" \
    --format mp4 \
    --work-dir ./noir_project

# Individual phases (for manual control)
./run.sh research "film noir lighting techniques"
./run.sh script --from-research research.json --duration 30 --use-create-story
./run.sh build-tools --script script.json
./run.sh generate --tools ./tools --script script.json --style "cinematic"
./run.sh assemble --assets ./assets --output movie.mp4 --format mp4
./run.sh learn --project-dir ./movie_project

CLI Commands

create

Full orchestrated workflow through all phases.

./run.sh create PROMPT [OPTIONS]
  --output, -o       Output file (default: movie.mp4)
  --work-dir, -w     Working directory (default: ./movie_project)
  --duration, -d     Target duration in seconds (default: 30)
  --style, -s        Visual style (e.g., 'cinematic', 'film noir')
  --format, -f       Output format: mp4 or html (default: mp4)
  --store-learnings  Store learnings in memory (default: true)
  --skip-research    Skip research phase if research.json exists
  --skip-casting     Skip casting phase (no identity packs)

research

Library-first research: checks Horus's memory and ingested content before external search.

./run.sh research TOPIC [OPTIONS]
  --output, -o       Output file (default: research.json)
  --skip-external    Only search library, skip external sources

script

Generate screenplay with scene breakdown. Integrates with /create-story.

./run.sh script [OPTIONS]
  --from-research, -r  Research JSON file (required)
  --prompt, -p         Override topic from research
  --duration, -d       Target duration in seconds
  --use-create-story   Use /create-story skill for screenplay
  --model, -m          LLM model (default: chimera)
  --output, -o         Output file (default: script.json)

build-tools

Generate custom tools in Docker sandbox.

./run.sh build-tools [OPTIONS]
  --script, -s       Script JSON file (required)
  --output-dir, -o   Output directory (default: ./tools)
  --skip-docker      Use host instead of Docker sandbox

generate

Create images, video, and audio assets.

./run.sh generate [OPTIONS]
  --tools, -t        Tools directory (default: ./tools)
  --script, -s       Script JSON file (required)
  --output-dir, -o   Assets output directory (default: ./assets)
  --style            Visual style to apply

assemble

Combine assets into final output.

./run.sh assemble [OPTIONS]
  --assets, -a       Assets directory (required)
  --output, -o       Output file/directory (required)
  --format, -f       Output format: mp4 or html (default: mp4)
  --fps              Frames per second for MP4 (default: 24)

learn

Store filmmaking insights in memory after a project.

./run.sh learn [OPTIONS]
  --project-dir, -p  Project directory (required)
  --scope            Memory scope (default: horus-filmmaking)
  --dry-run          Show learnings without storing

study

Pre-phase: Learn filmmaking topics BEFORE creating movies. Targeted /dogpile with internal (memory) + external (web) search, then stores via /memory learn.

./run.sh study TOPIC [OPTIONS]
  --scope            Memory scope (default: horus-filmmaking)
  --deep/--quick     Deep research (dogpile) vs quick (YouTube search)
  --list-topics      Show suggested filmmaking topics

# Examples:
./run.sh study "cinematography lighting techniques" --deep
./run.sh study "camera framing composition" --deep
./run.sh study --list-topics

study-all

Comprehensive learning session - studies all core filmmaking topics.

./run.sh study-all [OPTIONS]
  --scope            Memory scope (default: horus-filmmaking)

Output Formats

MP4 Video

Standard video file, playable anywhere.

Interactive HTML

Web-based experience with:

• Frame-by-frame navigation
• Audio controls
• Scene metadata viewer

Shot Specification (HorusShotSpec v0.1)

HorusShotSpec is a YAML-based shot specification format that replaces KSML for video generation.

Schema Overview

shot_id: "ACT1_SC02_SHOT03"
prompt:
  text: "A tense noir interrogation in a dim room. Slow dolly push toward suspect."
  negative: "text overlays, watermarks, shaky camera"
duration_s: 8                    # Valid: 4, 8, 16 seconds
aspect_ratio: "16:9"             # Valid: 16:9, 9:16, 1:1
resolution: "1080p"              # Valid: 720p, 1080p
references:
  subject_images:
    - path: "./assets/detective.png"
      weight: 0.7
controls:
  seed: 42
  safety: "default"
renderer:
  name: "veo"
  model: "veo-3.1-generate-preview"
metadata:
  scene: "SC02"
  act: "ACT1"
  sequence_order: 3

Compilation

The shot_compiler module validates and compiles YAML to Veo API JSON:

from core.shot_compiler import compile_yaml_to_veo_json

veo_request = compile_yaml_to_veo_json(yaml_content)
# → Returns dict ready for Veo API

Validation Rules

Migration from KSML

DEPRECATED: KSML is deprecated in favor of HorusShotSpec YAML. See docs/KSML_TO_YAML_MIGRATION.md for migration guide.

Quick comparison:

Available Skills

Horus has access to all skills in .pi/skills/:

Free/Open-Source Tools

Video Model Selection Guide

Choose video model based on your GPU VRAM and use case. VRAM figures include 3-5GB headroom for pipeline overhead (ComfyUI/loader/audio), batch=1, FP8/FP4 where noted.

Safe Defaults (RTX A5000 24GB)

Model: LTX-2 19B FP8
Resolution: 720p
Clip length: 10s
Batch size: 1
Seed: fixed
Audio: on

If runtime VRAM >22GB or instability occurs: lower resolution to 540p, disable audio, or shorten clips. Avoid parallel jobs on 24GB.

Model Characteristics

Recommendation:

• Use LTX-2 19B FP8 for production work with audio sync and camera controls
• Use WAN 2.2 for silent films or when audio isn't needed (higher visual quality for same VRAM)
• Fallback to Mochi for maximum quality or CogVideoX for compatibility

LTX-2: Recommended Video Model

LTX-2 is a 19B parameter DiT-based audio-video foundation model.

Model Variants:

FP8 Compatibility: Requires compatible CUDA/cuDNN/PyTorch builds. Follow LTX-Video docs for driver requirements.

Key Features:

• Synchronized Audio-Video Generation: Generates coherent audio + video together
• Camera Controls: Dolly, jib, static shots with natural camera motion
• IC-LoRA: Style transformations (anime, sketch, etc.) with ~1GB VRAM
• Keyframe Interpolation: Morphing between keyframes
• Pose/Depth/Canny Controls: Precise composition control (Canny edge detection)
• Text-to-Video and Image-to-Video: Both workflows supported

ComfyUI Templates:

Installation:

# ComfyUI (recommended)
# Install "LTX-Video" from ComfyUI Manager
# Templates appear automatically

# Or standalone
pip install ltx-video

ComfyUI VRAM Optimization Flags:

# Reserve VRAM for other operations (prevents OOM during generation)
python -m main --reserve-vram 5

# Low VRAM mode - offloads to system RAM (slower but prevents OOM)
python -m main --lowvram

# Weight streaming - NVIDIA/ComfyUI collaboration for 256GB RAM systems
# Automatically offloads model weights to system RAM when VRAM exhausted

Additional Resources:

• ComfyUI_LTX-2_VRAM_Memory_Management - Nodes for long videos on consumer GPUs

Camera Control Reference (LTX-2)

LTX-2 supports cinematic camera movements via prompt keywords:

Example Prompts:

# Dramatic reveal
"Dolly in slowly to a detective examining evidence, noir lighting, static hold on face"

# Action sequence
"Tracking shot following runner through city streets, handheld, dynamic"

# Interview setup
"Static medium shot, subject centered, shallow depth of field, jib down to hands"

Combining Movements:

"Jib up while dolly out, revealing vast landscape, golden hour, cinematic"

WAN 2.2: Silent Film Alternative

WAN 2.2 is a 14B parameter model optimized for visual quality without audio:

Best For:

• Silent films and art cinema
• German Expressionism era aesthetics (Nosferatu, Metropolis, Cabinet of Dr. Caligari)
• High visual fidelity when audio isn't needed
• Projects where audio will be added separately

Comparison to LTX-2:

When to Choose WAN 2.2:

• Creating silent films with intertitles
• German Expressionism homages
• Music videos where audio is pre-recorded
• Art films with separate sound design

Practical Notes: Seed control recommended for stable multi-shot outputs. 720p preferred on 24GB for consistent speeds.

Performance Expectations

Video generation is compute-intensive. Plan for overnight batch processing rather than real-time iteration.

Local Generation Times (RTX A5000, 24GB VRAM)

Notes:

• Timings based on Alex Ziskind's benchmarks (RTX 5080) with +15-25% buffer for A5000
• Audio synchronization adds ~10-15% time vs video-only runs
• IO/storage affects throughput; prefer local NVMe, avoid network mounts

Realistic Workflow

For a 2-minute film (12 x 10-second clips):

• Generation time: ~42-54 min (LTX-2, 720p) to ~60-72 min (WAN 2.2)
• With retakes and iterations: 2-4 hours
• Full production with assembly: overnight task

Recommendation: Queue video generation as overnight background tasks. Use /task-monitor to track progress.

# Example: Run generation overnight
./run.sh generate --script script.json --output-dir ./assets &
# Check progress next morning

RunPod for Large Tasks

Use /ops-runpod when local generation would cause OOM errors.

When to Use RunPod

OOM Threshold Guidance (A5000 24GB):

• LTX-2 FP8: 1080p clips over ~12-15s may OOM with audio; use 720p, shorten clips, or disable audio
• Control nets (pose/depth/canny) and multiple LoRAs increase memory; enable selectively
• Monitor runtime VRAM; keep ≤22GB to avoid instability

RunPod Workflow

# Provision GPU for large task
/ops-runpod provision --gpu a100-40gb --task "LTX-2 BF16 generation"

# Run generation on RunPod
/ops-runpod run --script generate.sh

# Download results and terminate
/ops-runpod download --output ./assets
/ops-runpod terminate

RunPod GPU Options:

• BF16/full precision: A100 40-80GB, H100 (required)
• FP8/FP4 tasks: L40S 48GB, A10G 24GB (cheaper alternatives)

Cost Consideration: RunPod charges by the hour. For overnight tasks, local generation is more cost-effective. Consider spot/preemptible instances for savings.

Troubleshooting & Fallbacks

OOM Mitigation:

1. Reduce resolution (720p → 540p)
2. Shorten clip length
3. Set batch=1
4. Switch FP mode (BF16 → FP8 → FP4)
5. Disable audio
6. Split long clips into segments

Stability:

• Fix seed for reproducibility
• Avoid parallel jobs on 24GB
• Reduce control nets and LoRA stacks

Fallback Path: If LTX-2 fails, switch to WAN 2.2 (video-only) or CogVideoX; add audio separately in post.

Memory Integration

After each movie, stores:

• Successful prompts
• Working tool code
• Technique insights
• Concept relationships

Scope: horus-filmmaking

Workflow Patterns (from Nobody & The Computer)

Multi-Model Collaboration

Different AI models handle different creative aspects, inspired by "Bach x Coltrane x Kuti x Takemitsu":

• Model A (Claude): Structure, composition, narrative arc
• Model B (GPT): Improvisation, dialogue, variation
• Model C (Grok): Energy, rhythm, pacing
• Model D (DeepSeek): Texture, atmosphere, silence

Each model builds on previous work. Constraints: 100 words max per turn for focused output.

Critique Loop

From "A.I.thoven" sessions - "roast the piece with love":

1. Generate initial draft
2. Critique constructively (what works, what doesn't)
3. Iterate based on feedback
4. Repeat until satisfied

Iteration Speed

Use LTX-2 Distilled for rapid iterations during creative exploration. Use LTX-2 13B for production with camera controls and audio sync. Fallback to Mochi for maximum quality when camera control isn't needed.

Example Session

Horus: I want to create a mockumentary about AI learning to paint.

[RESEARCH] Searching for documentary interview techniques, AI art history...
[SCRIPT] Breaking into 5 scenes: intro, discovery, struggle, breakthrough, reflection
[BUILD TOOLS] Writing code for interview framing effect, paint brush animation...
[GENERATE] Creating 45 frames, 3 audio tracks, 2 voice segments...
[ASSEMBLE] Combining into 2-minute video with transitions...
[LEARN] Storing 8 insights in memory for future films.

Output: ai_painter_mockumentary.mp4 (2:14)

Dependencies

• Docker (for isolated code execution)
• FFmpeg (video processing)
• Python 3.11+ (orchestrator)
• GPU recommended (for Stable Diffusion, video models)