ComfyUI Workflow Builder
Translates natural language requests into executable ComfyUI workflow JSON. Always validates against inventory before generating.
Workflow Generation Process
Step 1: Understand the Request
Parse the user's intent into:
- Output type: Image, video, or audio
- Source material: Text-only, reference image(s), existing video
- Identity method: None, zero-shot (InstantID/PuLID), LoRA, Kontext
- Quality level: Draft (fast iteration) vs production (maximum quality)
- Special requirements: ControlNet, inpainting, upscaling, lip-sync
Step 2: Check Inventory
Read state/inventory.json to determine:
- Available checkpoints → select best match for task
- Available identity models → determine which methods are possible
- Available ControlNet models → enable pose/depth control if available
- Custom nodes installed → verify all required nodes exist
- VRAM available → optimize settings accordingly
Step 3: Select Pipeline Pattern
Based on request + inventory, choose from:
| Pattern |
When |
Key Nodes |
| Text-to-Image |
Simple generation |
Checkpoint → CLIP → KSampler → VAE |
| Identity-Preserved Image |
Character consistency |
+ InstantID/PuLID/IP-Adapter |
| LoRA Character |
Trained character |
+ LoRA Loader |
| Image-to-Video (Wan) |
High-quality video |
Diffusion Model → Wan I2V → Video Combine |
| Image-to-Video (AnimateDiff) |
Fast video, motion control |
+ AnimateDiff Loader + Motion LoRAs |
| Talking Head |
Character speaks |
Image → Video → Voice → Lip-Sync |
| Upscale |
Enhance resolution |
Image → UltimateSDUpscale → Save |
| Inpainting |
Edit regions |
Image + Mask → Inpaint Model → KSampler |
Step 4: Generate Workflow JSON
ComfyUI workflow format:
{
"{node_id}": {
"class_type": "{NodeClassName}",
"inputs": {
"{param_name}": "{value}",
"{connected_param}": ["{source_node_id}", {output_index}]
}
}
}
Rules:
- Node IDs are strings (typically "1", "2", "3"...)
- Connected inputs use array format:
["source_node_id", output_index]
- Output index is 0-based integer
- Filenames must match exactly what's in inventory
- Seed values: use random large integer or fixed for reproducibility
Step 5: Validate
Before presenting to user:
- Every
class_type exists in inventory's node list
- Every model filename exists in inventory's model list
- All required connections are present (no dangling inputs)
- VRAM estimate doesn't exceed available VRAM
- Resolution is compatible with chosen model (512 for SD1.5, 1024 for SDXL/FLUX)
Step 6: Output
If online mode: Queue via comfyui-api skill
If offline mode: Save JSON to projects/{project}/workflows/ with descriptive name
Workflow Templates
Basic Text-to-Image (FLUX)
{
"1": {
"class_type": "LoadCheckpoint",
"inputs": {"ckpt_name": "flux1-dev.safetensors"}
},
"2": {
"class_type": "CLIPTextEncode",
"inputs": {"text": "{positive_prompt}", "clip": ["1", 1]}
},
"3": {
"class_type": "CLIPTextEncode",
"inputs": {"text": "{negative_prompt}", "clip": ["1", 1]}
},
"4": {
"class_type": "EmptyLatentImage",
"inputs": {"width": 1024, "height": 1024, "batch_size": 1}
},
"5": {
"class_type": "KSampler",
"inputs": {
"seed": 42,
"steps": 25,
"cfg": 3.5,
"sampler_name": "euler",
"scheduler": "normal",
"denoise": 1.0,
"model": ["1", 0],
"positive": ["2", 0],
"negative": ["3", 0],
"latent_image": ["4", 0]
}
},
"6": {
"class_type": "VAEDecode",
"inputs": {"samples": ["5", 0], "vae": ["1", 2]}
},
"7": {
"class_type": "SaveImage",
"inputs": {"filename_prefix": "output", "images": ["6", 0]}
}
}
With Identity Preservation (InstantID + IP-Adapter)
Extends basic template by adding:
- Load reference image node
- InstantID Model Loader + Apply InstantID
- IPAdapter Unified Loader + Apply IPAdapter
- FaceDetailer post-processing
See references/workflows.md for complete node settings.
Video Generation (Wan I2V)
Uses different loader chain:
- Load Diffusion Model (not LoadCheckpoint)
- Wan I2V Conditioning
- EmptySD3LatentImage (with frame count)
- Video Combine (VHS)
See references/workflows.md Workflow 4 for complete settings.
VRAM Estimation
| Component |
Approximate VRAM |
| FLUX FP16 |
16GB |
| FLUX FP8 |
8GB |
| SDXL |
6GB |
| SD1.5 |
4GB |
| InstantID |
+4GB |
| IP-Adapter |
+2GB |
| ControlNet (each) |
+1.5GB |
| Wan 14B |
20GB |
| Wan 1.3B |
5GB |
| AnimateDiff |
+3GB |
| FaceDetailer |
+2GB |
Common Mistakes to Avoid
- Wrong output index: CheckpointLoader outputs
[model, clip, vae] at indices [0, 1, 2]
- CFG too high for InstantID: Use 4-5, not default 7-8
- Wrong resolution for model: FLUX/SDXL=1024, SD1.5=512
- Missing VAE: FLUX needs explicit VAE (
ae.safetensors)
- Wrong model in wrong loader: Diffusion models need
LoadDiffusionModel, not LoadCheckpoint
Reference Files
references/workflows.md - Detailed node-by-node templates
references/models.md - Model files and paths
references/prompt-templates.md - Model-specific prompts
state/inventory.json - Current inventory cache
1---
2name: comfyui-workflow-builder
3description: Generate, build, create, or design ComfyUI workflow JSON from natural language descriptions. Produces valid node graphs with correct class_types, connections, output indices, and model-appropriate settings. Handles txt2img, img2img, inpainting, ControlNet, LoRA stacking, upscaling, and face detailing pipelines. Does NOT cover ComfyUI installation, custom node development, Python scripting, model training, hardware advice, or architectural explanations.
4user-invocable: true
5metadata:
6 source: https://skillmd.com/skills/mckruz/comfyui-workflow-builder
7 openclaw:
8 emoji: 🔧
9 os:
10 - darwin
11 - linux
12 - win32
13 requires:
14 anyBins:
15 - curl
16 - wget
17 primaryEnv: COMFYUI_URL
18---
19
20# ComfyUI Workflow Builder
21
22Translates natural language requests into executable ComfyUI workflow JSON. Always validates against inventory before generating.
23
24## Workflow Generation Process
25
26### Step 1: Understand the Request
27
28Parse the user's intent into:
29- **Output type**: Image, video, or audio
30- **Source material**: Text-only, reference image(s), existing video
31- **Identity method**: None, zero-shot (InstantID/PuLID), LoRA, Kontext
32- **Quality level**: Draft (fast iteration) vs production (maximum quality)
33- **Special requirements**: ControlNet, inpainting, upscaling, lip-sync
34
35### Step 2: Check Inventory
36
37Read `state/inventory.json` to determine:
38- Available checkpoints → select best match for task
39- Available identity models → determine which methods are possible
40- Available ControlNet models → enable pose/depth control if available
41- Custom nodes installed → verify all required nodes exist
42- VRAM available → optimize settings accordingly
43
44### Step 3: Select Pipeline Pattern
45
46Based on request + inventory, choose from:
47
48| Pattern | When | Key Nodes |
49|---------|------|-----------|
50| Text-to-Image | Simple generation | Checkpoint → CLIP → KSampler → VAE |
51| Identity-Preserved Image | Character consistency | + InstantID/PuLID/IP-Adapter |
52| LoRA Character | Trained character | + LoRA Loader |
53| Image-to-Video (Wan) | High-quality video | Diffusion Model → Wan I2V → Video Combine |
54| Image-to-Video (AnimateDiff) | Fast video, motion control | + AnimateDiff Loader + Motion LoRAs |
55| Talking Head | Character speaks | Image → Video → Voice → Lip-Sync |
56| Upscale | Enhance resolution | Image → UltimateSDUpscale → Save |
57| Inpainting | Edit regions | Image + Mask → Inpaint Model → KSampler |
58
59### Step 4: Generate Workflow JSON
60
61**ComfyUI workflow format:**
62
63```json
64{
65 "{node_id}": {
66 "class_type": "{NodeClassName}",
67 "inputs": {
68 "{param_name}": "{value}",
69 "{connected_param}": ["{source_node_id}", {output_index}]
70 }
71 }
72}
73```
74
75**Rules:**
76- Node IDs are strings (typically "1", "2", "3"...)
77- Connected inputs use array format: `["source_node_id", output_index]`
78- Output index is 0-based integer
79- Filenames must match exactly what's in inventory
80- Seed values: use random large integer or fixed for reproducibility
81
82### Step 5: Validate
83
84Before presenting to user:
85
861. Every `class_type` exists in inventory's node list
872. Every model filename exists in inventory's model list
883. All required connections are present (no dangling inputs)
894. VRAM estimate doesn't exceed available VRAM
905. Resolution is compatible with chosen model (512 for SD1.5, 1024 for SDXL/FLUX)
91
92### Step 6: Output
93
94**If online mode**: Queue via `comfyui-api` skill
95**If offline mode**: Save JSON to `projects/{project}/workflows/` with descriptive name
96
97## Workflow Templates
98
99### Basic Text-to-Image (FLUX)
100
101```json
102{
103 "1": {
104 "class_type": "LoadCheckpoint",
105 "inputs": {"ckpt_name": "flux1-dev.safetensors"}
106 },
107 "2": {
108 "class_type": "CLIPTextEncode",
109 "inputs": {"text": "{positive_prompt}", "clip": ["1", 1]}
110 },
111 "3": {
112 "class_type": "CLIPTextEncode",
113 "inputs": {"text": "{negative_prompt}", "clip": ["1", 1]}
114 },
115 "4": {
116 "class_type": "EmptyLatentImage",
117 "inputs": {"width": 1024, "height": 1024, "batch_size": 1}
118 },
119 "5": {
120 "class_type": "KSampler",
121 "inputs": {
122 "seed": 42,
123 "steps": 25,
124 "cfg": 3.5,
125 "sampler_name": "euler",
126 "scheduler": "normal",
127 "denoise": 1.0,
128 "model": ["1", 0],
129 "positive": ["2", 0],
130 "negative": ["3", 0],
131 "latent_image": ["4", 0]
132 }
133 },
134 "6": {
135 "class_type": "VAEDecode",
136 "inputs": {"samples": ["5", 0], "vae": ["1", 2]}
137 },
138 "7": {
139 "class_type": "SaveImage",
140 "inputs": {"filename_prefix": "output", "images": ["6", 0]}
141 }
142}
143```
144
145### With Identity Preservation (InstantID + IP-Adapter)
146
147Extends basic template by adding:
148- Load reference image node
149- InstantID Model Loader + Apply InstantID
150- IPAdapter Unified Loader + Apply IPAdapter
151- FaceDetailer post-processing
152
153See `references/workflows.md` for complete node settings.
154
155### Video Generation (Wan I2V)
156
157Uses different loader chain:
158- Load Diffusion Model (not LoadCheckpoint)
159- Wan I2V Conditioning
160- EmptySD3LatentImage (with frame count)
161- Video Combine (VHS)
162
163See `references/workflows.md` Workflow 4 for complete settings.
164
165## VRAM Estimation
166
167| Component | Approximate VRAM |
168|-----------|-----------------|
169| FLUX FP16 | 16GB |
170| FLUX FP8 | 8GB |
171| SDXL | 6GB |
172| SD1.5 | 4GB |
173| InstantID | +4GB |
174| IP-Adapter | +2GB |
175| ControlNet (each) | +1.5GB |
176| Wan 14B | 20GB |
177| Wan 1.3B | 5GB |
178| AnimateDiff | +3GB |
179| FaceDetailer | +2GB |
180
181## Common Mistakes to Avoid
182
1831. **Wrong output index**: CheckpointLoader outputs `[model, clip, vae]` at indices `[0, 1, 2]`
1842. **CFG too high for InstantID**: Use 4-5, not default 7-8
1853. **Wrong resolution for model**: FLUX/SDXL=1024, SD1.5=512
1864. **Missing VAE**: FLUX needs explicit VAE (`ae.safetensors`)
1875. **Wrong model in wrong loader**: Diffusion models need `LoadDiffusionModel`, not `LoadCheckpoint`
188
189## Reference Files
190
191- `references/workflows.md` - Detailed node-by-node templates
192- `references/models.md` - Model files and paths
193- `references/prompt-templates.md` - Model-specific prompts
194- `state/inventory.json` - Current inventory cache