Agent N Video Pipeline (Fully Local on M5)
Produce a character-consistent "Agent N presents a message" video entirely on
the 16 GB Apple-Silicon M5 executor. Port of the cloud Herald pipeline
(hermes_build/herald_pipeline.sh + build_final_video.py) that replaces both
cloud legs (grok-imagine image-to-video, volcengine lip-sync) with ONE local
MLX leg (LTX-2.3 A2V, native audio-to-video).
One-command usage
# in the repo: scripts/ dir
./herald_pipeline_m5.sh --spec <spec.json> --anchor agentn_anchor_v1.png
# optional: --m5 192.168.212.46 (default) --voice <path> (skip TTS)
Spec JSON = single source of truth, same schema as the cloud Herald version:
{ "script": "...", "prompt": "...", "out": "...", "duration": N, "voice": "..." }.
Default spec (research-receipts) is embedded.
Stages (all local)
- TTS — Kokoro sid 10 (mature female voice) via
kokoro_tts.py
(sherpa-onnx; model ~/.hermes/kokoro/kokoro-multi-lang-v1_0/, download the
kokoro-multi-lang-v1_0.tar.bz2 bundle from k2-fsa/sherpa-onnx
tts-models release if purged). Override with --voice to skip.
- Talking render + lip sync in ONE leg —
build_final_video_m5.py queues
LTX-2.3 MLX A2V on the M5 via ComfyUI REST: anchor image + audio →
speaking video. 512×512, frames = duration×24, cfg 3.0, low_ram,
dgrauet/ltx-2.3-mlx-q4. Then poll + pull the mp4.
- Karaoke captions —
karaoke_burn.py (PressStart2P, word-synced to the
TRUE script). Unchanged from the cloud pipeline.
- Trim — ffmpeg to voice duration +2.5 s.
- QA gate —
qa_review.py (deterministic). Call separately in the repo.
- X caption —
x_caption.py (mentions @NousResearch/@Teknium).
Segmented mode (recommended for >10 s messages)
Render in 3–8 s chunks, each re-anchored on the Agent N image (identity lock,
no cumulative drift), then concat (each segment carries its synced audio) →
karaoke on the stitched video. queue_segments.py does this: split voice by
sentence boundaries, queue one A2V per segment, stitch with ffmpeg concat.
The anchor
The anchor image IS the identity lock — sheet-derived, front-facing, ≥512²,
mouth unobstructed, no burned-in text. Never drive identity from prompt alone.
Performance facts (measured on M5 16GB, Q4, 512×512)
- ~16 s/frame on MPS with low_ram block-streaming (Q4). 2 s clip ≈ 13 min.
- 5 s ≈ 32 min, 8 s ≈ 51 min, 22 s ≈ 2.3 h single-pass → use segmented mode.
- Q8@512×768 swaps (5.5 GB swap) — do not use on 16 GB. Q4 + 512×512 + low_ram
is the reliable envelope.
Pitfalls already climbed (do not re-climb)
- LTX-MLX-A2V node install: ltx-2-mlx@v0.14.15 pyproject has a readme-path bug
on subdir installs; clone repo, remove the
readme line from both subpackage
pyprojects, pip install from local clone.
- Needs
transformers < 5.10 (mlx-lm 0.31 conflict).
- ltx-core-mlx requires ffmpeg on PATH: symlink
ffmpeg/ffprobe into the
ComfyUI venv bin or export PATH (user-local brew ~/homebrew/opt/ffmpeg/bin).
- ComfyUI must launch with
PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0.
- Flat/calm TTS → near-static mouth (node README). Expressive takes articulate.
- First A2V run downloads the Q8/Q4 model (~8–10 GB) — subsequent runs cached.
- kokoro_tts.py writes long audio with
array('h'), NOT struct.pack("%dh",*s)
(unpacking ~500k args breaks on long scripts); ensure output dir exists.
Env gate before running
- M5 ComfyUI headless:
curl -s http://M5:8188/system_stats
- Anchor staged in M5
ComfyUI-Shared/input/
- Kokoro model dir present; venv
~/.venvs/kokoro/bin/python has sherpa_onnx
- LM Studio / llama-server off (frees ~5 GB)
1---2name: agent-n-video-pipeline3description: >4 Use this skill when producing a fully-local "Agent N presents a message" video on5 the 16GB Apple-Silicon M5 executor: Kokoro mature TTS → LTX-2.3 MLX A2V6 talking render with native lip sync → karaoke captions → trim → QA → X caption.7 Replaces the old cloud Herald pipeline (grok-imagine + volcengine) with zero8 cloud cost. Activates for Herald release videos, Agent N announcements, and9 newsletter video messages.10version: 1.0.011author: Hermes Agent12license: MIT13compatibility: >14 Cross-platform: Hermes Agent (default profile orchestrates, M5 executor15 renders locally). Scripts are plain bash/python + ffmpeg + ComfyUI REST.16platforms:17 - hermes-agent18tags:19 - video20 - agent-n21 - ltx22 - local-generation23 - talking-head24 - herald25metadata:26 source: https://skillmd.com/skills/jpeetz/agent-n-video-pipeline27---2829# Agent N Video Pipeline (Fully Local on M5)3031Produce a character-consistent "Agent N presents a message" video entirely on32the 16 GB Apple-Silicon M5 executor. Port of the cloud Herald pipeline33(`hermes_build/herald_pipeline.sh` + `build_final_video.py`) that replaces both34cloud legs (grok-imagine image-to-video, volcengine lip-sync) with ONE local35MLX leg (LTX-2.3 A2V, native audio-to-video).3637## One-command usage3839```bash40# in the repo: scripts/ dir41./herald_pipeline_m5.sh --spec <spec.json> --anchor agentn_anchor_v1.png42# optional: --m5 192.168.212.46 (default) --voice <path> (skip TTS)43```4445Spec JSON = single source of truth, same schema as the cloud Herald version:46`{ "script": "...", "prompt": "...", "out": "...", "duration": N, "voice": "..." }`.47Default spec (research-receipts) is embedded.4849## Stages (all local)50511. **TTS** — Kokoro sid 10 (mature female voice) via `kokoro_tts.py`52 (sherpa-onnx; model `~/.hermes/kokoro/kokoro-multi-lang-v1_0/`, download the53 `kokoro-multi-lang-v1_0.tar.bz2` bundle from k2-fsa/sherpa-onnx54 `tts-models` release if purged). Override with `--voice` to skip.552. **Talking render + lip sync in ONE leg** — `build_final_video_m5.py` queues56 LTX-2.3 MLX A2V on the M5 via ComfyUI REST: anchor image + audio →57 speaking video. 512×512, frames = duration×24, cfg 3.0, low_ram,58 `dgrauet/ltx-2.3-mlx-q4`. Then poll + pull the mp4.593. **Karaoke captions** — `karaoke_burn.py` (PressStart2P, word-synced to the60 TRUE script). Unchanged from the cloud pipeline.614. **Trim** — ffmpeg to voice duration +2.5 s.625. **QA gate** — `qa_review.py` (deterministic). Call separately in the repo.636. **X caption** — `x_caption.py` (mentions @NousResearch/@Teknium).6465## Segmented mode (recommended for >10 s messages)6667Render in 3–8 s chunks, each re-anchored on the Agent N image (identity lock,68no cumulative drift), then concat (each segment carries its synced audio) →69karaoke on the stitched video. `queue_segments.py` does this: split voice by70sentence boundaries, queue one A2V per segment, stitch with ffmpeg concat.7172## The anchor7374The anchor image IS the identity lock — sheet-derived, front-facing, ≥512²,75mouth unobstructed, no burned-in text. Never drive identity from prompt alone.7677## Performance facts (measured on M5 16GB, Q4, 512×512)7879- ~16 s/frame on MPS with low_ram block-streaming (Q4). 2 s clip ≈ 13 min.80- 5 s ≈ 32 min, 8 s ≈ 51 min, 22 s ≈ 2.3 h single-pass → use segmented mode.81- Q8@512×768 swaps (5.5 GB swap) — do not use on 16 GB. Q4 + 512×512 + low_ram82 is the reliable envelope.8384## Pitfalls already climbed (do not re-climb)8586- LTX-MLX-A2V node install: ltx-2-mlx@v0.14.15 pyproject has a readme-path bug87 on subdir installs; clone repo, remove the `readme` line from both subpackage88 pyprojects, pip install from local clone.89- Needs `transformers < 5.10` (mlx-lm 0.31 conflict).90- ltx-core-mlx requires ffmpeg on PATH: symlink `ffmpeg`/`ffprobe` into the91 ComfyUI venv bin or export PATH (user-local brew `~/homebrew/opt/ffmpeg/bin`).92- ComfyUI must launch with `PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0`.93- Flat/calm TTS → near-static mouth (node README). Expressive takes articulate.94- First A2V run downloads the Q8/Q4 model (~8–10 GB) — subsequent runs cached.95- kokoro_tts.py writes long audio with `array('h')`, NOT `struct.pack("%dh",*s)`96 (unpacking ~500k args breaks on long scripts); ensure output dir exists.9798## Env gate before running99100- M5 ComfyUI headless: `curl -s http://M5:8188/system_stats`101- Anchor staged in M5 `ComfyUI-Shared/input/`102- Kokoro model dir present; venv `~/.venvs/kokoro/bin/python` has sherpa_onnx103- LM Studio / llama-server off (frees ~5 GB)