Parallel Streams

parallel-streams: build a project in parallel terminals

The orchestrator (you) freezes the contracts, streams (headless claude -p loops in their own terminals) build independently, you integrate at the end. Why this shape: streams cannot see each other. Every shared file becomes a merge accident, every vague definition-of-done becomes a false "finished". The upfront contracts are the actual success factor.

Status: what is tested, what is not

The PowerShell runner (assets/windows/) is battle-tested on Windows: it has run real overnight builds with multiple parallel streams, including limit pauses, resumes, steering and takeovers. The POSIX port (assets/posix/) is a faithful line-by-line port but has NOT been executed on macOS/Linux yet; it is syntax-checked only. Review it and run one short throwaway stream before you trust it with a night.

Orchestrator triage: what belongs in a stream, what does NOT

Streams are normal claude -p sessions with your full user config (skills, agents, CLAUDE.md, user-scope MCP servers, web tools, gh CLI). Everything file-based, stateless or API-key-driven runs in a stream exactly as it would for you.

The one class that does not work headless: anything that needs an interactive login, an OAuth popup, or your live browser session, because nobody is sitting there to click it. Keep those parts in your own interactive terminal: browser MCPs on logged-in sites, OAuth-gated connectors, and above all browser-based visual QA. A stream that still hits such a source must mark it as "needs manual login/QA" and report it back instead of forcing it.

Rule of thumb when splitting: file-based / API key / stateless goes to a stream. Login / OAuth / browser / visual QA stays with you.

Phase 1: freeze the contracts (BEFORE any stream starts)

1. Scan the current state (a read-only explore agent): interfaces, schema, entry points, test conventions.
2. Write SPEC-<project>.md into the repo. Shared deltas (schema, registries, contracts) you implement YOURSELF up front. Hub files (schema, app entry/router registry, central UI shell, install/deploy) belong to the orchestrator; no stream touches them.
3. Ownership matrix into the SPEC: every file belongs to exactly ONE stream. If two streams need the same file, the file moves to the orchestrator and the streams deliver integration snippets.
4. Pre-wire registries defensively (e.g. a router list with try/except imports) so streams only have to add new modules.

Phase 2: stream briefs

Per stream, write <repo>/.streams/prompts/stream-<X>.md containing: role · "read the SPEC COMPLETELY first" · scope (reference SPEC paragraphs instead of duplicating them) · DoD checklist (own tests + the WHOLE suite green + build green + ownership respected) · conventions: update .streams/status/<X>.progress.md every iteration (the resume anchor), write <X>.integration.md (exact wiring instructions for the orchestrator), create <X>.done ONLY when every DoD item is met · no git · no new dependencies · forward slashes in shell paths (on Windows, backslash paths in Git Bash create junk folders) · do not touch deploy/production.

Mandatory block in EVERY brief: "re-entry after a context reset" (reality before memory). The runner deliberately discards the session on a limit sleep (context rotation, see phase 3), so every fresh start has only the brief plus progress.md. progress.md can lag minutes behind reality (the limit hits mid-work). So put this into each brief verbatim: before ANY building, (1) read progress.md, (2) check reality (git log/git status or an artifact listing with timestamps), (3) NEVER recreate or blindly overwrite existing work; take it as done and continue exactly where the work REALLY stops, (4) correct progress.md to the true state immediately. Duplicated work is a bug, not diligence. (Proven in a real overnight run: three streams rotated three times each, zero duplicated work.)

Bonus backlog against finishing early (optional, recommended for runs with time or budget slack): below the DoD, add a priority-sorted bonus list. Semantics then: <X>.done only once the DoD is met AND the bonus list is worked off or the rest is documented as [BLOCKED: <reason>]. The bonus items do NOT extend the DoD itself, otherwise optional points block the finish signal.

Plan for steer latency: steers only take effect at the next iteration start, and a hung process never pulls one (see the watchdog in phase 3). Everything known before launch belongs in the brief; the steer file is only the top-up channel, with latency.

Phase 3: pre-flight + launch

1. Keep the machine awake. Sleep kills every stream mid-flight.
   • Windows: powercfg /change standby-timeout-ac 0 and powercfg /change hibernate-timeout-ac 0
   • macOS: caffeinate -dimsu -w <runner-pid> (per runner; stops with it), or one caffeinate -dimsu for the whole night
   • Linux: wrap the launch in systemd-inhibit --what=sleep --who=parallel-streams --why="overnight streams" sh .streams/launch.sh ...
2. Ideally start with a fresh subscription window; N streams drain it roughly N times faster. A tight window is fine (the runner pauses itself), it only costs wall-clock time. Hard cap: 3 streams in parallel. Each stream is its own node process plus MCP servers plus builds; more pushes a typical machine into paging. With more than 3 packages, queue them: the watcher (step 7) starts the next stream when a slot frees up (done/ended). Runners run at low priority (BelowNormal on Windows, niceness 10 on POSIX) so the machine stays usable.
3. Delete old .streams/status/ files AND any leftover .streams/steer/<X>.md for the stream names you reuse; otherwise resume picks up a foreign session and the first iteration swallows a stale steer note from the previous run.
4. Copy the assets once (they live next to this SKILL.md):
   • Windows: assets/windows/run-stream.ps1 to <repo>/.streams/run-stream.ps1, assets/windows/takeover-stream.ps1 to <repo>/.streams/takeover.ps1, assets/windows/launch-stream.ps1 to <repo>/.streams/launch.ps1
   • macOS/Linux: assets/posix/run-stream.sh, assets/posix/takeover.sh, assets/posix/launch.sh to <repo>/.streams/run-stream.sh, .streams/takeover.sh, .streams/launch.sh
5. Stream permissions: default acceptEdits. Streams see agents, skills and all user-scope MCP servers automatically, but tool CALLS beyond file edits (MCP, Bash, WebFetch, ...) are permission-gated. Prefer granting via the runner parameter (-AllowedTools / --allowed-tools): ONE comma-separated string that becomes --allowedTools. Verified working: exact names like mcp__firecrawl__firecrawl_scrape, wildcards like mcp__firecrawl__*, and rules like Bash(pytest:*). The CLI flag works regardless of workspace trust. The alternative (an allowlist in <repo>/.claude/settings.json) only works in trusted workspaces (opened interactively once + trust accepted); otherwise it is silently IGNORED headless ("workspace has not been trusted" in the stream log). The deny side exists too (-DisallowedTools / --disallowed-tools, ONE comma-separated string that becomes --disallowedTools): deny beats allow, which makes it the right tool to hard-fence guardrail runs, e.g. Bash(git push:*),Bash(gh api:*),Bash(curl:*) for a "nothing goes online" night. Mind the prefix trap in both lists: allow/deny rules match literal command prefixes, so git -C <path> status does NOT match Bash(git status:*). Either tell briefs to cd instead of using git -C, or cut the rules deliberately wider. Full permission bypass ONLY if the user explicitly approves it for THIS project.
6. Launch each stream through the launch helper. Passing the model is MANDATORY: claude -p does not inherit the launching session's model, it silently uses the CLI default.
   • Windows (helper exists to prevent focus stealing; details in the appendix): pwsh -NoProfile -File "<repo>\.streams\launch.ps1" -Repo "<repo>" -Stream <X> -Model <model-id> Optionally -AllowedTools "mcp__foo__*,...". Returns a status object (WindowFound/Minimized/InTaskbar/UserFocusKept).
   • macOS/Linux: sh <repo>/.streams/launch.sh --repo <repo> --stream <X> --model <model-id> Optionally --allowed-tools "mcp__foo__*,...". Starts the runner via nohup ... & (survives the terminal), prints pid, state and log paths. Then check the PID locks (.streams/status/*.pid).
7. Watcher as a background task (plain shell, NOT a model: deterministic beats model-driven here): a poll loop (5 min) over done markers + live runner PIDs; exit when all are done (you get woken) or when all runners are dead with an incomplete done set. Extra signal: .streams/status/<X>.state (starting|claude-running|limit-sleep|paused|done|ended). With a queue (>3 packages) the watcher launches the next stream through the same launch helper. Dead unfinished runners it relaunches with FULL arguments (cap the attempts, e.g. 5 with a 10-minute backoff; never relaunch state 'ended'). Hang watchdog (mandatory in the watcher): state claude-running AND .streams/status/<X>.heartbeat older than ~15 min usually means the runner hangs in a frozen claude pipeline; one long, silent tool call produces the same signature, so the threshold is a heuristic. A hung runner cannot rescue itself, it is blocking in the pipe; in one real overnight run a stream hung like that for close to four hours and never pulled its steer. Then: kill the process tree (the PID is in <X>.pid; on Windows taskkill /PID <pid> /T /F, on POSIX kill the group), relaunch like a dead runner. Important: do NOT put the done check before the hang check; a stream can have written its done marker and still hang (eating RAM, never pulling a steer again). Revive pattern (documented option): an early-finished stream can be brought back: delete <X>.done, the watcher relaunches (state 'done' does not count as 'ended'), the runner resumes or starts fresh, and a pre-written steer file delivers the new assignment. No need to touch the running watcher for that; a small standalone helper (waits for a free slot, mandatory streams take precedence, the 3-stream cap stays hard) is safer than live-editing a running script.

The runner itself handles limit tolerance: exit!=0 or session shorter than 120s means 15 min pause; a normal session end without done marker resumes immediately (--resume <session_id>, context survives); a PID lock prevents double runners; hard deadline at the next occurrence of the deadline hour (parameter, default 9:00). Context rotation (default ON): when entering the limit sleep the runner discards the session id, so the next iteration starts FRESH with brief + progress.md. Why: permanent resuming grows the context monotonically (the model gets worse and every turn pays for the full context), and after the 15-minute pause the prompt cache is cold anyway, so the fresh start costs nothing extra. The counterpart is the mandatory re-entry block in the brief (phase 2), otherwise duplicated work looms. Disable only deliberately (-NoContextRotate / --no-context-rotate, e.g. very short runs without progress.md discipline). The runner also writes <X>.heartbeat from the stream renderer (throttled to every 30s on Windows, every event in the POSIX port) for the hang watchdog.

Live view & intervening (hybrid: the autopilot stays intact)

The engine stays headless (claude -p + auto-resume). On Windows the runner renders the stream-json output TUI-style into the (minimized) window: iteration separator, model + session line, thinking lines, tool calls, agent spawns, results, a final "done · s · tokens" line, plus a pinned status bar (model, elapsed, state, tokens, iteration). If the output is not a real terminal, everything falls back to plain text automatically. The POSIX runner always writes the plain-text form; follow it with tail -f .streams/logs/stream-<X>.log. The log shows the same content either way. Context-% is not available headless; it only becomes visible in takeover mode.

Steer and takeover both take effect at the next iteration start, never inside a running iteration. Two intervention paths, neither breaks auto-resume:

• Steer (async, gentle): write a note into <repo>/.streams/steer/<X>.md. The runner appends it to the prompt at the NEXT iteration start with priority and moves the file to .streams/steer/applied/ (one-shot delivery). Latency: up to one iteration. Right for course corrections that do not need to be instant.
• Takeover (live, interactive): run the takeover script in a real terminal:
  • Windows: pwsh -NoProfile -File "<repo>\.streams\takeover.ps1" -Repo "<repo>" -Stream <X>
  • macOS/Linux: sh <repo>/.streams/takeover.sh --repo <repo> --stream <X> It sets a pause flag, waits until the runner is not mid-iteration (avoiding two instances on the same session), then opens the same session interactively (claude --resume: full TUI including context display, you can type along). After you exit: session-id handback (resume can fork) + flag removed, and the runner continues automatically on the NEW state.

Phase 4: integration, QA & iteration (orchestrator)

1. NEVER trust done markers blindly. Re-verify the DoD yourself (suite, build, spot checks); in real runs, half the streams reported "finished" too early at least once.
2. Read every <X>.integration.md, wire the hub files, run the whole suite + build, then deploy.
3. QA is YOUR job, not the streams' (browser QA does not run headless). Physically click through every built app/UI in a real browser. Scripted E2E or curl never replaces this (service workers, cookies, auth popups and caches are only visible in a browser). Default timing: batched, once the watcher reports that ALL streams are done. A baseline look before the start helps spot regressions. Mid-run QA only when one stream is much shorter than the others: check its done marker ad hoc and test it early while the long ones keep running. Only report "finished" after green browser QA.
4. Split-again loop: if QA finds a defect that is bigger than a 30-second fix, treat it as a new work package: brief fix streams and run them through the same engine (overnight again if needed) instead of grinding through everything yourself. Small findings you fix directly. Build, QA and fix stay one loop that keeps turning unattended.
5. Streams that are not finished at the deadline: read their progress.md, finish the rest yourself or restart the stream with a tighter scope.

Appendix: Windows specifics

• Focus stealing: Windows Terminal always activates newly created windows/tabs; neither Task Scheduler nor ForegroundLockTimeout prevents it (hard-tested). The launch helper therefore starts the runner as a HIDDEN conhost window and then reveals it minimized into the taskbar via SW_SHOWMINNOACTIVE: zero focus loss, at the cost of one taskbar icon per stream instead of shared tabs.
• conhost ANSI rendering: conhost only interprets ANSI with VT processing enabled. One-time, user-scope, reversible: Set-ItemProperty 'HKCU:\Console' VirtualTerminalLevel 1 -Type DWord Without it, conhost shows raw escape codes. The runner additionally enables ENABLE_VIRTUAL_TERMINAL_PROCESSING on its own handle and uses font-robust glyphs (● └) that render in Consolas as well as Cascadia.
• -ClaudeArgs as an array does not survive wt/Start-Process launches (pwsh -File drops the follow-up tokens); that is why extra tool grants travel as ONE comma-separated -AllowedTools string.