Comfy QA

comfy-qa

Automated QA demo video generator for Comfy-Org's public-facing products. Given a target (PR, issue, feature, or product URL), it generates narrated browser demo videos showing real user workflows.

What this project does

Input:  PR / issue / feature doc / product URL
  ↓
Phase 1: Probe → discover site structure (headless browser)
Phase 2: Story → generate .comfy-qa/02-stories/<product>.story.md
Phase 3: Spec  → generate .comfy-qa/03-spec/<product>.spec.ts
Phase 4: Video → render .comfy-qa/04-videos/<product>.mp4
  ↓
Output: Narrated MP4 with TTS, cursor HUD, subtitles

Quick start

# Install
bun install
bun run prepare   # sync demowright submodule

# Generate a demo for a product
# 1. Write or generate a story in .comfy-qa/02-stories/
# 2. Write or generate a spec in .comfy-qa/03-spec/
# 3. Run the spec
bunx playwright test .comfy-qa/03-spec/<product>.spec.ts

# Run all demos
bunx playwright test

# Evaluate output videos with Gemini
bun tmp/review-videos.ts

Project structure

src/
├── agent/
│   ├── demo-research.ts   ← explores sites, generates stories
│   ├── demo-editor.ts     ← post-processes raw recordings
│   ├── browser-agent.ts   ← AI-driven browser automation
│   ├── orchestrator.ts    ← full QA pipeline (fetch → research → record → report)
│   └── research.ts        ← PR/issue analysis agent
├── cli.ts                 ← CLI entry point (comfy-qa)
└── utils/                 ← GitHub API, URL parsing, ComfyUI helpers

demo/
├── fixtures/
│   ├── fixture.ts         ← Playwright fixture (HUD + Gemini TTS + audio)
│   └── mux-reporter.ts    ← video + audio → MP4 muxer

.claude/skills/
├── demo-spec-author/SKILL.md  ← spec authoring rules and patterns

.comfy-qa/                 ← ALL generated output (gitignored)
├── spec/                  ← generated Playwright spec files
├── stories/               ← generated story markdown files
├── checklists/            ← generated feature checklists (YAML)
├── videos/                ← rendered MP4 videos
└── .tmp/                  ← intermediate Playwright artifacts

Key principles

1. User journeys, not museum tours

Every demo must show a complete user workflow — search, click, navigate, type, copy — not just hover over elements. See .claude/skills/demo-spec-author/SKILL.md for the full spec-authoring guide.

2. Segment classification

Each video segment is classified as NAVIGATE (page transitions via setup callback), INTERACT (clicks, typing), or OBSERVE (hover/scroll). At least 50% of segments must be NAVIGATE or INTERACT.

3. Coverage scoring

Demo quality is measured by enumerating all CRUD operations on the target site and calculating coverage as a ratio (e.g., 13/22 = 59%). Never use simple pass/fail.

4. Zero idle frames

Every segment must have continuous cursor movement during narration. No segment where the mouse sits still while the narrator talks.

Environment variables (.env.local)

GEMINI_API_KEY=...          # Required: TTS narration (Gemini 2.5 Flash)
CLOUD_USERNAME=...          # Optional: cloud.comfy.org login
CLOUD_PASSWORD=...
GOOGLE_USERNAME=...         # Optional: registry.comfy.org email login
GOOGLE_PASSWORD=...

Submodule: demowright

lib/demowright provides cursor HUD overlay, TTS integration, and video production via createVideoScript(). After pulling, run bun run prepare to sync it.

CLI

comfy-qa pr <owner/repo#number>     # QA a pull request
comfy-qa issue <owner/repo#number>  # QA an issue
comfy-qa full <owner/repo>          # Batch QA for a repo