Architecture
Workflows live in workflows/ with components and flows:
workflows/
├── index.md # Inventory with tags
├── components/
│ ├── connections/ # Auth configs
│ ├── nodes/ # Reusable operations
│ └── triggers/ # Event sources
└── flows/{name}/
├── flow.md # Definition
├── config.yaml # Parameters
├── run.sh # Executable
├── state/ # Persistent between runs
└── logs/
Quick Reference
| Topic |
File |
| Directory layout, naming, formats |
structure.md |
| Data passing between nodes |
data-flow.md |
| Cursor, seen set, checkpoint |
state.md |
| Retry, rollback, idempotency |
errors.md |
| Connections, nodes, triggers |
components.md |
| Create, test, update, archive |
lifecycle.md |
Requirements
- jq — JSON processing
- yq — YAML config parsing
- curl — HTTP requests
- flock — Lock files to prevent concurrent runs
- Secrets in macOS Keychain (
security find-generic-password)
Data Storage
- Location:
workflows/ in workspace root
- State:
flows/{name}/state/ — cursor.json, seen.json, checkpoint.json
- Logs:
flows/{name}/logs/ — JSONL per run
- Data:
flows/{name}/data/ — intermediate files between nodes
Core Rules
1. Data Flow Pattern
Each node writes output to data/{NN}-{name}.json. Next node reads it.
curl ... > data/01-fetch.json
jq '...' data/01-fetch.json > data/02-filter.json
Breaking this pattern = nodes can't communicate.
2. Error Declaration
Every node in flow.md MUST declare:
- On error: retry(N) | fail | continue | alert
- On empty: skip | continue | fail
Undefined behavior = unpredictable workflow.
3. Lock Files
Prevent concurrent runs:
LOCKFILE="/tmp/workflow-${NAME}.lock"
exec 200>"$LOCKFILE"
flock -n 200 || exit 0
4. State Files
| File |
Purpose |
| cursor.json |
"Where did I leave off?" |
| seen.json |
"What have I processed?" |
| checkpoint.json |
"Multi-step recovery" |
5. Component Reuse
Before creating new connections/nodes/triggers:
ls workflows/components/connections/
ls workflows/components/nodes/
Use existing. Update "Workflows Using This" when adding.
Related: For LLM-driven multi-phase processes, see the cycle skill.
---
name: workflow
description: Build automated pipelines with reusable components, data flow between nodes, and state management.
---
## Architecture
Workflows live in `workflows/` with components and flows:
```
workflows/
├── index.md # Inventory with tags
├── components/
│ ├── connections/ # Auth configs
│ ├── nodes/ # Reusable operations
│ └── triggers/ # Event sources
└── flows/{name}/
├── flow.md # Definition
├── config.yaml # Parameters
├── run.sh # Executable
├── state/ # Persistent between runs
└── logs/
```
## Quick Reference
| Topic | File |
|-------|------|
| Directory layout, naming, formats | `structure.md` |
| Data passing between nodes | `data-flow.md` |
| Cursor, seen set, checkpoint | `state.md` |
| Retry, rollback, idempotency | `errors.md` |
| Connections, nodes, triggers | `components.md` |
| Create, test, update, archive | `lifecycle.md` |
## Requirements
- **jq** — JSON processing
- **yq** — YAML config parsing
- **curl** — HTTP requests
- **flock** — Lock files to prevent concurrent runs
- Secrets in macOS Keychain (`security find-generic-password`)
## Data Storage
- **Location:** `workflows/` in workspace root
- **State:** `flows/{name}/state/` — cursor.json, seen.json, checkpoint.json
- **Logs:** `flows/{name}/logs/` — JSONL per run
- **Data:** `flows/{name}/data/` — intermediate files between nodes
## Core Rules
### 1. Data Flow Pattern
Each node writes output to `data/{NN}-{name}.json`. Next node reads it.
```bash
curl ... > data/01-fetch.json
jq '...' data/01-fetch.json > data/02-filter.json
```
Breaking this pattern = nodes can't communicate.
### 2. Error Declaration
Every node in flow.md MUST declare:
- **On error:** retry(N) | fail | continue | alert
- **On empty:** skip | continue | fail
Undefined behavior = unpredictable workflow.
### 3. Lock Files
Prevent concurrent runs:
```bash
LOCKFILE="/tmp/workflow-${NAME}.lock"
exec 200>"$LOCKFILE"
flock -n 200 || exit 0
```
### 4. State Files
| File | Purpose |
|------|---------|
| cursor.json | "Where did I leave off?" |
| seen.json | "What have I processed?" |
| checkpoint.json | "Multi-step recovery" |
### 5. Component Reuse
Before creating new connections/nodes/triggers:
```bash
ls workflows/components/connections/
ls workflows/components/nodes/
```
Use existing. Update "Workflows Using This" when adding.
---
**Related:** For LLM-driven multi-phase processes, see the `cycle` skill.
