# Joelclaw > Operate the joelclaw event bus, gateway, observability, and agent loop infrastructure via the joelclaw CLI. Use for: sending events, checking runs, starting/monitoring/cancelling agent loops, debugging failed runs, checking health, restarting the worker, inspecting step traces, managing subscriptions, gateway operations, OTEL queries, semantic recall. Triggers: 'joelclaw', 'send an event', 'check inngest', 'start a loop', 'loop status', 'why did this fail', 'debug run', 'check worker', 'restart worker', 'runs', 'what failed', 'subscribe', 'gateway', 'otel', 'recall', or any Inngest/event-bus/agent-loop task. - Skill: `joelhooks-joelclaw/joelclaw` (Agent Skill) - Install (CLI): `npx skillmds@latest add joelhooks-joelclaw/joelclaw` - Raw SKILL.md: https://api.skillmd.com/api/skills/joelhooks-joelclaw/joelclaw/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: joelhooks (https://skillmd.com/u/joelhooks-joelclaw) - Updated: 2026-09-09 - Page: https://skillmd.com/skills/joelhooks-joelclaw/joelclaw --- # joelclaw — CLI & Event Bus The `joelclaw` CLI is the primary operator interface to the entire joelclaw system: event bus (Inngest), gateway, observability (OTEL), agent loops, subscriptions, and more. Built with `@effect/cli`, returns HATEOAS JSON envelopes. **If the CLI crashes, that's the highest priority fix.** Binary: `~/.bun/bin/joelclaw` Source: `~/Code/joelhooks/joelclaw/packages/cli/` Build: `bun build packages/cli/src/cli.ts --compile --outfile ~/.bun/bin/joelclaw` ## Architecture ``` ┌─ Colima VM (VZ framework, aarch64) ────────────────────────────────┐ │ Talos v1.12.4 → k8s v1.35.0 (single node, namespace: joelclaw) │ │ │ │ inngest-0 StatefulSet ports 8288 (API), 8289 (dash) │ │ redis-0 StatefulSet port 6379 │ │ typesense-0 StatefulSet port 8108 │ │ restate-0 StatefulSet ports 8080, 9070, 9071 │ │ system-bus-worker Deployment port 3111 (110+ functions) │ │ restate-worker Deployment port 9080 │ │ dkron-0 StatefulSet port 8080 │ │ minio-0 StatefulSet ports 9000, 9001 │ │ docs-api Deployment port 3838 │ │ livekit-server Deployment ports 7880, 7881 │ │ bluesky-pds Deployment port 3000 │ │ │ │ ⚠️ Inngest service named inngest-svc (not inngest) │ │ k8s auto-injects INNGEST_PORT env collision otherwise │ └─────────────────────────────────────────────────────────────────────┘ ↕ NodePort on localhost Gateway daemon (always-on pi session, Redis event bridge) NAS "three-body" (ASUSTOR, 10GbE NFS, 64TB RAID5 + NVMe cache) Vault ~/Vault (Obsidian, PARA method — ADRs, system log, contacts) ``` **Inngest event key**: read from `INNGEST_EVENT_KEY`; do not hardcode old dev keys. **k8s manifests**: `~/Code/joelhooks/joelclaw/k8s/` ## CLI Command Reference ### Health & Status ```bash joelclaw status # Health: server + worker + k8s pods joelclaw jobs status # Unified runtime workload snapshot (queue + Restate + Dkron + Inngest) joelclaw inngest status # Inngest server details joelclaw functions # List all 110+ registered functions joelclaw refresh # Force re-register with Inngest server ``` ### Send Events ```bash joelclaw send "event/name" --data '{"key":"value"}' joelclaw send "pipeline/video.requested" --data '{"url":"https://youtube.com/watch?v=XXX"}' joelclaw send "agent/story.start" --data '{"prdPath":"/abs/path/prd.json","storyId":"S-1"}' ``` ### View Runs ```bash joelclaw runs # Recent 10 joelclaw runs --count 20 --hours 24 # More runs, wider window joelclaw runs --status FAILED # Just failures joelclaw run # Step trace + errors for one run ``` ### View Events ```bash joelclaw events # Last 4 hours joelclaw events --prefix memory/ --hours 24 joelclaw events --prefix agent/ --hours 24 joelclaw events --count 50 --hours 48 ``` ### Logs ```bash joelclaw logs # Worker stdout (default 30 lines) joelclaw logs errors # Worker stderr (stack traces) joelclaw logs server # Inngest k8s pod logs joelclaw logs server -n 50 --grep error # Filtered server errors joelclaw logs worker --grep "observe" # Grep worker logs ``` ### Durable receipts and telemetry The legacy `joelclaw log write` / `slog` JSONL journal is retired. Emit/query structured runtime telemetry with `joelclaw otel`; record durable decisions and change receipts in Brain `.svx`. `joelclaw logs` remains the runtime log read/analyze surface. ### Secrets ```bash joelclaw secrets status joelclaw secrets lease --ttl 15m joelclaw secrets revoke # Revoke only the exact leases acquired by this task. joelclaw secrets audit --tail 50 joelclaw secrets env --dry-run ``` ### Notify ```bash joelclaw notify send "Worker restarted and healthy" --kind receipt joelclaw notify send "Immediate action required" --kind alert ``` `--priority` is deprecated compatibility metadata. Use `--kind`; the versioned routing table owns delivery. ### Capability Adapter Paths (ADR-0169 phase 4) `mail`, `otel`, `recall`, and `subscribe` now run through the CLI capability registry/adapter runtime while preserving their existing command UX and JSON envelopes. ### Deploy ```bash joelclaw deploy worker # dry-run deploy plan joelclaw deploy worker --restart --execute # execute worker sync deployment joelclaw deploy worker --restart --execute --force # force with active runs (disruptive) ``` ### Heal ```bash joelclaw heal list joelclaw heal run RUN_FAILED --phase fix --context '{"run-id":"01ABC"}' # dry-run joelclaw heal run RUN_FAILED --phase fix --context '{"run-id":"01ABC"}' --execute # execute ``` ### Gateway ```bash joelclaw gateway status # Gateway health + session info joelclaw gateway events # Recent gateway events joelclaw gateway test # Send test message through gateway joelclaw gateway restart # Restart gateway daemon joelclaw gateway stream # Live stream gateway events ``` ### Observer `observer` is the canonical joelclaw Observer target. Do not guess panes from the default Herdr session, and do not treat it as a hidden special case. Resolve and manage it through the CLI first: ```bash joelclaw observer status # Herdr protocol + Observer registry/session metadata joelclaw observer resolve observer # Canonical target → named Herdr session + pane joelclaw observer resolve dispatcher # Dispatcher pane in the Observer family joelclaw observer cycle # Dry-run safe named-session cycle plan joelclaw observer cycle --execute # Cycle only the named observer Herdr server, then rotate Observer ``` Use `joelclaw observer resolve observer` before scheduling or waking the Observer. If it fails with `OBSERVER_TARGET_UNHEALTHY`, run `joelclaw observer status`; if the Herdr protocol drifted, run `joelclaw observer cycle --execute`. `joelclaw wake ... --target observer` is the right wake target once resolve is healthy. ### Workload Rig ```bash joelclaw workload plan "intent" --stages-from stages.json --write-plan plan.json joelclaw workload run plan.json --stage --tool pi|codex|claude joelclaw workload dispatch plan.json --to BlueFox --from MaroonReef --send-mail joelclaw workload sandboxes list joelclaw workload sandboxes cleanup --expired --dry-run joelclaw workload sandboxes janitor ``` Runtime truth: - `workload run` admits into Redis, then Restate executes the stage through `dagOrchestrator` → `dagWorker` - `dagWorker` handlers: `shell`, `infer`, `microvm` - `infer` now runs `pi` inside the `restate-worker` k8s pod with mounted auth, skills, and identity - `--stages-from` is the explicit stage-DAG front door; dependency gates and cycle detection are real ### Async runtime monitoring in pi The loaded pi extension at `packages/pi-extensions/inngest-monitor/index.ts` now exposes `runtime_jobs_monitor` alongside `inngest_send` / `inngest_runs`. Use it when you want a pi session to keep watch on the ADR-0217 runtime substrate while you do other work: - `runtime_jobs_monitor {"action":"start","interval":5,"report":true}` — start background polling of `joelclaw jobs status` - `runtime_jobs_monitor {"action":"status"}` — inspect the latest runtime snapshot - `runtime_jobs_monitor {"action":"stop"}` — stop the poller and send a final follow-up summary The widget shows runtime health first (queue / Restate / Dkron / Inngest), then any followed Inngest runs underneath. Severity changes and meaningful workload-state changes emit OTEL and hidden follow-up messages for asynchronous report-back, so a healthy-but-now-held backlog does not stay silent. ### Observability (OTEL) ```bash joelclaw otel list --hours 1 # Recent telemetry events joelclaw otel search "error" --hours 24 # Search OTEL events joelclaw otel stats --hours 24 # Aggregate stats joelclaw otel emit "action.name" --source codex --component agent-loop --success # Emit event ``` ### System Knowledge (ADR-0199) ```bash joelclaw knowledge search "query" # Search system knowledge joelclaw knowledge search "query" --type adr # Filter by type (adr|skill|lesson|pattern|retro|failed_target) joelclaw knowledge sync # Re-index ADRs + skills from filesystem joelclaw knowledge clear-failed # Clear resolved failed targets ``` Brain/codebase patterns (browsable by agents): ```bash ls ~/Vault/system/brain/codebase/ # List established patterns cat ~/Vault/system/brain/codebase/.md # Read a specific pattern ``` ### Subscriptions (ADR-0127) ```bash joelclaw subscribe list # All feed subscriptions joelclaw subscribe add [--name N] # Add a feed joelclaw subscribe remove # Remove a feed joelclaw subscribe check [--url URL] # Check feeds for new items joelclaw subscribe summary # Summary of recent items ``` ### Agent Runtime Validation (ADR-0180) Use this exact smoke test when validating roster dispatch end-to-end: ```bash joelclaw agent list joelclaw agent run coder "reply with OK" --timeout 20 joelclaw event ``` Expected signal: - `agent list` includes builtin `coder`, `designer`, `ops` - `event` shows one `Agent Task Run` with `status: COMPLETED` - run output contains `{"status":"completed", ...}` Failure handling: - `Unknown agent roster entry: coder` means runtime drift, not prompt failure. - Deploy latest `system-bus-worker` - Restart host worker process - Re-run the same 3-step smoke - If Inngest API is unreachable (`localhost:8288`), recover local control-plane first (Colima/Talos), then retry validation. ### Brain-backed Recall ```bash joelclaw recall "query about past context" # Search disposable Brain + observation projections ``` Brain `.svx` pages remain canonical. The Typesense `observations` and `brain_graph_nodes` collections are rebuildable projections, not a second memory store. ### Discovery ```bash joelclaw discover "https://example.com" --context "why this is interesting" ``` ### Agent Loops ```bash # Start a loop joelclaw loop start --project ~/Code/joelhooks/joelclaw \ --goal "Implement feature X" \ --context ~/Vault/docs/decisions/0XXX.md \ --max-retries 2 # Start with existing PRD joelclaw loop start --project PATH --prd prd.json --max-retries 2 # Monitor joelclaw loop status joelclaw loop status -c # Compact: one line per story joelclaw loop status -v # Verbose: criteria, output paths joelclaw watch # Live: polls 15s, exits on completion joelclaw watch # Auto-detects active loop # Management joelclaw loop list # All loops in Redis joelclaw loop cancel # Stop + cleanup joelclaw loop nuke dead # Remove completed loops from Redis ``` ### Other Commands ```bash joelclaw sleep [on|off|status] # Sleep mode for gateway joelclaw note # Quick note to Vault joelclaw vault read # Resolve/read ADR/project/path refs joelclaw vault search # Search vault markdown joelclaw vault adr list # ADR inventory (optionally by status) joelclaw vault adr collisions # ADR number collision report joelclaw vault adr audit # ADR health + collision + index checks joelclaw vault adr rank # ADR NRC+novelty ranking joelclaw skills audit [--deep] # On-demand skill garden report joelclaw search # Full-text search joelclaw email [scan|triage] # Email operations joelclaw x [post|mentions] # X/Twitter operations joelclaw nas [status|health] # NAS operations joelclaw diagnose # System diagnosis joelclaw usage [--source router|agents|all] # Token usage + cost from ClickHouse joelclaw deploy worker [--restart] [--execute] joelclaw heal [list|run] joelclaw inngest sync-worker [--restart] # Worker lifecycle ``` For vault-heavy or ADR-gardening tasks, use the dedicated [`vault`](../vault/SKILL.md) skill. ### Output Modes Most commands support `--compact/-c` for plain text. Use compact for monitoring. JSON (default) returns HATEOAS envelopes with `next_actions`. ## Story Pipeline (ADR-0155) 3-stage pipeline: implement → prove → judge. Each story runs through the stages with Inngest durability. ```bash # Fire a single story joelclaw send agent/story.start -d '{ "prdPath": "/Users/joel/Code/joelhooks/joelclaw/prd.json", "storyId": "CFP-2" }' # ⚠️ ALWAYS use absolute path for prdPath — worker CWD is packages/system-bus/ ``` **PRD format** (Zod-validated): ```json { "name": "Project Name", "context": {}, "stories": [ { "id": "STORY-1", "title": "What to build", "description": "Details", "priority": 1, "acceptance": ["criterion 1", "criterion 2"], "files": ["path/to/relevant/file.ts"] } ] } ``` **Critical:** - `context` must be `{}` or object — NEVER null or string - Every story needs `priority` (number) - **NEVER set `retries: 0`** on Inngest functions — breaks restart safety (ADR-0156) ## Event Types ### Pipelines | Event | Chain | |-------|-------| | `pipeline/video.requested` | → video-download → transcript-process → content-summarize | | `pipeline/video.download` | legacy alias for `pipeline/video.requested` | | `pipeline/transcript.requested` | → transcript-process → content-summarize | | `pipeline/transcript.process` | legacy alias for `pipeline/transcript.requested` | | `content/summarize.requested` | → content-summarize | | `content/summarize` | legacy alias for `content/summarize.requested` | | `content/updated` | → content-sync (git commit vault changes) | | `docs/ingest` | → docs-ingest (PDF/markdown → vector store) | ### Memory The event-driven observation/reflect/promote chain is retired and no longer registered. Current ownership is: | Layer | Current path | |-------|--------------| | Evidence | Runtime adapters produce immutable accepted Runs | | Operational memory | One collector and one semantic worker project flowing observations and reflections | | Curated memory | Brain `.svx` pages | | Recall | Flowing reflections, flowing observations, and curated pages as separate lanes | | Raw history | Explicit `joelclaw sessions search` drill-down | ### Agent Loops | Event | Flow | |-------|------| | `agent/story.start` | → story-pipeline (implement → prove → judge) | | `agent/loop.started` | → plan → story pipeline → complete | | `agent/loop.story.passed` | → plan (next story) | | `agent/loop.story.failed` | → plan (retry or next) | | `agent/loop.completed` | → complete (merge-back + cleanup) | ### Gateway & Channels | Event | Purpose | |-------|---------| | `gateway/message.received` | Incoming message from any channel | | `gateway/heartbeat` | Gateway health ping | | `channel/telegram.callback` | Telegram callback queries | ### Subscriptions & Discovery | Event | Purpose | |-------|---------| | `discovery/noted` | URL/idea captured → enrichment pipeline | | `subscriptions/check` | Poll feeds for new items | ### System & Scheduled | Event | Purpose | |-------|---------| | `system/log` | System log entry | | `system/health.check` | Scheduled health monitoring | | `cron/daily-digest` | Morning digest generation | | `cron/check-email` | Periodic email scan | | `cron/check-calendar` | Calendar check | | `cron/nightly-maintenance` | Typesense + system maintenance | ### Notifications | Event | Source | |-------|--------| | `webhook/github` | GitHub webhook events | | `webhook/vercel` | Vercel deploy events | | `webhook/todoist` | Todoist webhook events | | `webhook/front` | Front webhook events | ## Debugging Failed Runs ```bash joelclaw runs --status FAILED # 1. Find the failure joelclaw run # 2. Step trace + inline errors joelclaw logs errors # 3. Worker stderr joelclaw logs server --grep error # 4. Inngest server errors joelclaw otel search "error" --hours 1 # 5. OTEL telemetry ``` ### Common Failure Patterns | Symptom | Cause | Fix | |---------|-------|-----| | Events accepted but functions never run | Inngest can't reach worker | `joelclaw refresh`, check worker pod | | "Unable to reach SDK URL" | Worker unreachable from cluster | Restart worker, `joelclaw refresh` | | Loop story SKIPPED | Tests/typecheck failed in worktree | Check attempt output | | Run stuck in RUNNING | Worker crashed mid-step | `joelclaw logs errors`, restart worker | | `INNGEST_PORT` env collision | k8s service named `inngest` | Service is `inngest-svc` — keep this | | Implement step killed on deploy | Worker restart killed in-flight step | ADR-0156: retries: 2 survives this | ### Stale RUNNING forensics (SDK unreachable ghosts) When `joelclaw runs --status RUNNING` shows old health jobs that never clear: 1. **Use the operator command first** - Preview: `joelclaw inngest sweep-stale-runs` - Apply (backup + transaction): `joelclaw inngest sweep-stale-runs --apply` 2. **Validate the symptom class** - `joelclaw run ` - Look for trace/finalization errors containing `Unable to reach SDK URL` or `EOF writing request to SDK`. 3. **Treat list vs detail disagreements as a known mask issue** - `runs` list can show stale metadata. - `run` detail + trace/history is the source of truth. 4. **Raw runtime DB edits are last resort only** - Inngest state is in k8s StatefulSet PVC: `inngest-0:/data/main.db`. - Backup first: `kubectl -n joelclaw exec inngest-0 -- sqlite3 /data/main.db '.backup /data/main.db.pre-sweep-.sqlite'`. 5. **Terminalize stale runs with full contract, not partial edits** - Insert missing `history.type='FunctionCancelled'` for stale runs. - Ensure `function_finishes` row exists. - Then set `trace_runs.status=500` (cancelled) for stale candidates. 6. **Verify after mutation** - `joelclaw run ` should resolve terminal state. - `joelclaw runs --status RUNNING` should only show genuinely active runs. Never mutate `main.db` without a point-in-time backup. ## Deploying Worker Changes Use the publish script — it handles build, push, k8s apply, and rollout: ```bash ~/Code/joelhooks/joelclaw/k8s/publish-system-bus-worker.sh ``` See the [sync-system-bus skill](../sync-system-bus/SKILL.md) for the full deploy workflow. ## Key Paths | What | Path | |------|------| | CLI source | `packages/cli/src/` | | CLI commands | `packages/cli/src/commands/` | | CLI binary | `~/.bun/bin/joelclaw` | | Worker source | `packages/system-bus/` | | Inngest functions | `packages/system-bus/src/inngest/functions/` | | Function index (host) | `packages/system-bus/src/inngest/functions/index.host.ts` | | Function index (cluster) | `packages/system-bus/src/inngest/functions/index.cluster.ts` | | Inference utility | `packages/system-bus/src/lib/inference.ts` | | Gateway source | `packages/gateway/` | | k8s manifests | `k8s/` | | Deploy script | `k8s/publish-system-bus-worker.sh` | | ADRs | `~/Vault/docs/decisions/` | | Telemetry | `joelclaw otel` (`otel_events` in Typesense/ClickHouse) | | Durable receipts | Brain `.svx` in the relevant registered root | | Loop attempt output | `/tmp/agent-loop/{loopId}/{storyId}-{attempt}.out` | ## Building the CLI ```bash cd ~/Code/joelhooks/joelclaw bun build packages/cli/src/cli.ts --compile --outfile ~/.bun/bin/joelclaw ``` **Test after every change:** ```bash joelclaw status joelclaw send --help joelclaw runs --count 1 ``` CLI commands are in `packages/cli/src/commands/`, one file per command. Follow the [cli-design skill](../cli-design/SKILL.md). Heavy deps must be lazy-loaded — top-level import crashes are unacceptable.