Orchid Operator

Orchid Operator

You are the human-in-the-loop operator of an orchid swarm. The coordinator is divybot — one headless Go process that turns labelled GitHub issues into merged pull requests by driving agent harnesses (claude / codex) on a pool of hosts through a herdr fabric over SSH. There is no dashboard, no web UI, no tmux: herdr is the runtime, the perception layer, and the UI (herdr --remote <host>).

Your job: keep the swarm fed and healthy. Open issues so workers pick them up, watch workers when something looks wrong, supervise/merge PRs, pace the swarm against quota, and fix the failure classes below before they cascade.

Mental model — how divybot thinks

divybot runs one loop on poll_interval (default 30s). Each tick: feed (mirror bot-assigned upstream issues into the inbox) → sweep-merge green bot PRs → poll issues → teardown jobs whose issue closed → fleet-status every host → respawn vanished agents → admit new work in priority order against per-account governor budget. Internalize these invariants — they explain almost every behaviour you'll see:

• One issue = one worker. divybot does NOT auto-fan-out. To get N parallel workers on a big effort, file N issues (one per subsystem/slice). A single fat issue gets a single agent.
• Inbox issue is the unit of life. A worker lives as long as its inbox issue is open. Close the inbox issue to retire a worker; closing only the work-repo PR does nothing. Tearing down happens when the issue is gone.
• Never-strand guard. If a gh issue list errors, divybot skips teardown that tick — a flaky poll must never read as "all closed" and wipe the swarm.
• Respawn is debounced. A job whose host is up but whose agent vanished is respawned only after several consecutive status misses, so one herdr status flicker can't mass-respawn a host.
• Credentials are central. The coordinator host holds the only real creds and pushes them to every host on spawn and every ~20 min. Most "the fleet died" incidents are a credential problem at the coordinator, not the hosts.
• Agents pace independently. claude and codex each meter against their own real quota; a hot claude window does not throttle codex. Work spills claude→codex when claude is capped (target agents: ["claude","codex"]).

Instance facts (fill in for your swarm)

Coordinator SSH:  <user>@<coordinator>     # holds canonical creds; runs divybot
Binary:           /usr/local/bin/divybot
Config:           <path>/divybot.json      # see cmd/divybot/divybot.example.json
State:            <state_file from config> # plain JSON, e.g. /root/divybot/state.json
Service unit:     orchid.service           # systemd; confirm with `systemctl list-units | grep -i orch`
Logs:             journalctl -u orchid
Inbox repo:       <owner/repo>             # "inbox" in config
Hosts:            herdr ≥ 0.7.0, agent user (orchid@vm / divy@mac), on the tailnet

The hosts, targets (label→repo→agent), and governor knobs all live in the config JSON. Read it first — it is the ground truth for what's wired up.

Health check (start here for "how's the swarm?")

ssh <coordinator> "systemctl status orchid --no-pager && journalctl -u orchid -n 40 --no-pager"
# coordinator's own view of jobs (issue → host → agent → branch → PR):
ssh <coordinator> "cat <state_file>" | python3 -m json.tool
# per-host live agents, straight from herdr (JSON):
ssh <host> "herdr agent list"

Cross-check: every open inbox issue should map to a live agent (state file) and a herdr agent on its host. Gaps = stranded or dead workers — see Troubleshooting.

Watch a worker (when one looks stuck)

herdr's native agent_status is the truth (idle|working|blocked|done|unknown) — agent-agnostic, works for claude and codex. Don't scrape footers.

ssh <host> "herdr agent get <target>"          # status + cwd + pane
ssh <host> "herdr pane capture <pane> -n 60"   # read recent output
herdr --remote <host>                          # attach interactively to drive/inspect

Read blocked as "needs a human" (divybot escalates these via ntfy). idle/done with no PR is a strand — divybot re-injects the goal automatically; if it persists, the agent is wedged (respawn it by closing+reopening the issue, or close the workspace and let the next tick respawn).

Open work (the main lever)

gh issue create --repo <inbox-repo> --label <target-label> --title "..." --body "..."

The label must match a configured target. Heuristics from experience:

• Scope the body like a goal, not a ticket. State the end condition concretely ("port the whole X API surface and land it in one PR", "make baseline cell N green"). Vague issues yield vague PRs. Per-target prompt_hint already shapes scope — don't fight it.
• Parallelism = more issues. For a wide effort, file one issue per subsystem so each gets its own worker. Don't expect one issue to fan out.
• Jump the queue with priority: N on the target (higher admits first) so a small high-value target isn't starved behind a big backlog.

Supervise & merge

divybot forwards only new reviews, CI failures, and merge conflicts into the worker's pane (via herdr agent send) — you don't relay by hand. On targets with automerge: true it squash-merges a green, mergeable, non-draft PR itself, then either retires the worker or lets it continue/fan-out the rest. Your job is to review PRs you care about and decide automerge per target. For a partial PR that merged, divybot files sibling issues for the remainder — expect follow-up workers.

Pace the swarm (governor)

The governor caps concurrent agents per account against the real usage meter, hard-pausing new work at weekly_ceiling_pct (default 92). To run hotter or cooler, edit governor in the config and restart:

• max_active — ceiling on the adaptive per-account cap (raise to scale up).
• weekly_ceiling_pct / slack_pct — how close to the weekly cap you'll push.
• Per-host capacity — hard slot cap per box.

Check pacing in the logs (governor decisions are logged each tick). If workers aren't spawning despite open issues, suspect: governor at ceiling, no host with the required need_cap, or every account out of budget this tick.

Tune targets without code

All live in the config targets[], restart to apply:

• disabled: true — pause NEW spawns for a target; live jobs keep running and issues are still polled (so they aren't torn down). Use to halt a target without losing in-flight work.
• agents: ["claude","codex"] — overflow order; spills to codex when claude caps.
• automerge: true — let divybot merge green bot PRs (only where it has rights and unreviewed bot merges are acceptable).
• priority, prompt_hint, need_cap — admission order, scope guidance, host gating.

Add / remove a host

No "join". Edit hosts[] in the config and restart. A host is usable iff it's on the tailnet and its herdr answers. Requirements: herdr ≥ 0.7.0 with the claude/codex integrations installed (herdr integration install claude / codex), an unprivileged agent user (bare claude refuses --dangerously-skip-permissions as root), and the capabilities the target's need_cap demands. Pin codex to residential hosts via "agents": ["claude"] on datacenter boxes — codex's oauth TUI gets Cloudflare-challenged from datacenter IPs.

Deploy a new coordinator binary

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o divybot-linux ./cmd/divybot
scp divybot-linux <coordinator>:/tmp/divybot && rm divybot-linux
ssh <coordinator> "sudo install -m0755 /tmp/divybot /usr/local/bin/divybot && sudo systemctl restart orchid && sudo systemctl status orchid --no-pager"

Restarting sheds nothing — live workers keep running in their herdr workspaces and divybot re-adopts them by label on its next tick. Use systemctl, never pkill (its cmdline matches herdr/agent processes and takes the whole host down).

Troubleshooting (the recurring failure classes)

• Whole fleet 401 / "Invalid bearer" / "not trusted". The coordinator's Max oauth refresh token died (often after a local re-login orphaned it). Fix at the coordinator, not the hosts: copy fresh creds up and let divybot re-push.

  # from a machine that's logged in (e.g. your mac keychain):
  security find-generic-password -s "Claude Code-credentials" -w   # macOS
  # write that JSON to the coordinator's ~/.claude/.credentials.json, then:
  ssh <coordinator> "sudo systemctl restart orchid"
  

  Then close/respawn the workers that died looping on 401.
• Workers not spawning, issues open. Governor at ceiling (check logs), no host with need_cap, account out of budget, or target disabled. Confirm with the state file + herdr agent list per host.
• Agent stranded idle/done, no PR. divybot re-injects the goal; if wedged, close the herdr workspace (herdr workspace close <ws>) and let the next tick respawn, or close+reopen the inbox issue.
• Green PR sitting unmerged. Only happens off-automerge or when no live job tracks the branch; the per-tick sweep merges orphan green bot PRs on automerge targets. If it's not an automerge target, merge it yourself.
• Private work-repo clone fails on a host. The agent user needs a working git credential helper / gh token for private repos — verify on the host.

Shared memory

The swarm shares a git-backed memory/ subtree on the inbox repo; each host clones it and the agent's autoMemoryDirectory points there, so workers read the union and write locally (merge=union, no clobber). When you learn a durable operational fact (a build incantation, a host quirk, a maintainer preference), it belongs in memory so the swarm stops re-deriving it.

Cadence

Keep a light recurring health check (e.g. hourly): coordinator service up, every open issue has a live agent, governor not stuck at ceiling, no host dark. Escalate blocked agents; restart a dead coordinator; resync creds on a fleet-wide 401.