# Mini App > Use when adding, removing, password-protecting, or troubleshooting a mini-app served by the hermes-config mini-app router (Caddy + PM2 + auth sidecar + Tailscale Serve/Funnel) on a fleet machine. Covers install, the Caddy route pattern, the auth sidecar conventions, exposing apps publicly via Funnel, Hermes dashboards behind a password, and the recurring pitfalls (Tailscale "serve reset" wars, the PM2 $HOME trap, funnel-eligible ports, strip-prefix requirements). - Skill: `technickai/mini-app` (Agent Skill, multi-file: 12 files) - Install (CLI): `npx skillmds@latest add technickai/mini-app` - Raw SKILL.md: https://api.skillmd.com/api/skills/technickai/mini-app/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: TechNickAI (https://skillmd.com/u/technickai) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/technickai/mini-app --- # Mini-App **Mission:** Operate the hermes-config mini-app router on this machine — the lightweight stack that exposes one or more named "mini-apps" at clean URL paths on a single Tailscale HTTPS host, with optional per-app password gating. One front door, many apps, no cloud. A mini-app is any process that binds to a localhost port and serves HTTP — a Node service, a Python FastAPI app, a Hermes dashboard, a webhook receiver. The app-router fronts them all with Caddy and gives each one: - A clean path (`https:////`) - Optional password protection via an Express auth sidecar - HTTPS for free via Tailscale Serve (and optional public exposure via Funnel) - PM2 supervision so it survives crashes and reboots The router stack lives upstream in [hermes-config](https://github.com/TechNickAI/hermes-config) under `devops/app-router/`. This skill is the operator playbook — what to do once it's installed. ## When to use Load this skill when you (a Hermes fleet agent) are asked to: - Add a new mini-app to this host's router (Hermes dashboard, webhook receiver, status page, anything that binds to localhost) - Remove or rename an existing mini-app - Add or change a password on a gated mini-app - Expose a mini-app publicly via Tailscale Funnel - Diagnose a 502 / 404 / auth-loop on a mini-app - Reload Caddy after editing the Caddyfile - Restore Tailscale Serve after another tool wiped it - Verify the front door is up after a reboot or upgrade **Don't use for:** writing the mini-app's code itself (that's a normal app), or changing the auth sidecar's source (that's an upstream PR in hermes-config). ## Stack at a glance ``` Internet (optional) │ Tailscale Funnel on:443 (only if you've opted in per-host) ▼ Tailnet host: ..ts.net │ ▼ 127.0.0.1:8080 (Caddy — declarative, hot-reloadable) ├── /auth/* → auth sidecar (Node + Express, JWT-style cookies) ├── /health → "ok" 200 ├── /hooks/* → optional bearer-injected webhook proxy ├── //* → mini-app on a localhost port ├── //* → mini-app on a localhost port (password-gated) └── / → static welcome page ``` **Single sources of truth on this machine** (after install): | File | What it controls | | ---------------------------------------- | ---------------------------------------------------------- | | `~/mini-apps/ecosystem.config.js` | PM2 process list + ALL env vars (incl. auth passwords) | | `~/mini-apps//Caddyfile` | Path → upstream routing | | `~/mini-apps//tailscale-serve.json` | Public/tailnet exposure (huJSON; compiled by apply script) | > **`` is `router/` or `_registry/` — check before you edit.** The current > installer creates `router/`; hosts installed before that carry `_registry/`. Both > are in the wild, so resolve it once at the start of any task and reuse it. > > Resolve the operator's real home first — under a Hermes tool environment `$HOME` > may point at the profile home, where `~/mini-apps` does not exist even though the > router is running (see "The router dir may have been renamed" below): > > ```bash > # anchor on the caddy BINARY path so this cannot match your own shell. > # -ww keeps long argv from being width-clipped on platforms that do that. > APPS=$(ps -eww -o args= | grep '^/[^ ]*caddy run' \ > | sed -n 's#.*--config \(.*\)/[^/]*/Caddyfile.*#\1#p' | head -1) > APPS=${APPS:-$(ls -d /Users/*/mini-apps /home/*/mini-apps 2>/dev/null | head -1)} > REG=$(ls -d "$APPS"/router "$APPS"/_registry 2>/dev/null | head -1) > echo "apps=$APPS reg=$REG" > ``` > > An empty `REG` with a running Caddy means the probe missed, NOT that the router is > uninstalled — fall back to the ground-truth search below before concluding anything. > > Examples below write `$APPS` and `$REG`. On a host you have already identified you > can substitute the literal paths. Never run ad-hoc `tailscale serve …` commands. Edit the JSON, run the apply script. ## First-time install on this machine ```bash brew install caddy # macOS — use the equivalent on Linux npm install -g pm2 cd ~/src/hermes-config # clone if you don't have it bash devops/app-router/scripts/install.sh ``` The installer copies templates into `~/mini-apps/`, renders the Caddyfile with the right paths, installs auth-service deps, and stages the launchd plist (macOS only). Re-running is safe; `--force` overwrites existing files. **Then by hand.** A fresh install from the current installer creates `router/`, so set `REG` before following these steps: ```bash REG=$(ls -d ~/mini-apps/router ~/mini-apps/_registry 2>/dev/null | head -1) ``` 1. Edit `~/mini-apps/ecosystem.config.js`: - Set `AUTH_SECRET` to `openssl rand -hex 32` (one per machine) - Add `APP_PASSWORD_` / `APP_TITLE_` / `APP_DESC_` for any gated apps 2. Edit `$REG/Caddyfile` to declare each app's route 3. Edit `$REG/tailscale-serve.json` if you want a non-default serve layout (default exposes Caddy on `:443`) **Start everything under PM2:** ```bash pm2 start ~/mini-apps/ecosystem.config.js pm2 start /opt/homebrew/bin/caddy --name caddy --interpreter none -- \ run --config "$REG/Caddyfile" --adapter caddyfile pm2 save pm2 startup # paste the printed sudo command — needed for resurrect-on-reboot ``` **Apply Tailscale Serve:** ```bash "$REG"/apply-tailscale-serve.sh # macOS launchd that replays on login: launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.hermes.mini-app-router-serve.plist ``` **If another tool on this host manages Tailscale Serve, disable that first.** The usual failure is a second process running `tailscale serve reset` on startup, which wipes these routes. Hermes' own gateway has no Tailscale integration and does not touch serve config, so a Hermes-only host needs no opt-out. A legacy agent runtime on the same machine may still own it — check what else drives serve: ```bash tailscale serve status # what is configured now grep -rl "tailscale serve" ~/Library/LaunchAgents \ ~/.config/systemd/user 2>/dev/null # who else replays it ``` ## Adding a mini-app Three edits, then reload. Pick a slug (`^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$`) and a free localhost port (convention: apps start at `3001`). **1. PM2 ecosystem entry** — append to `apps` in `ecosystem.config.js`: ```js { name: "my-app", script: "./my-app/server.js", // or absolute path, or a python command via interpreter cwd: "~/mini-apps", env: { PORT: 3001 }, }, ``` If password-gating, also add to the `auth-service` block's `env`: ```js APP_PASSWORD_MY_APP: "the-password", APP_TITLE_MY_APP: "My App", APP_DESC_MY_APP: "One-line description shown on the login form.", ``` The slug-to-env rule is uppercase + `-`→`_`. `my-app` ⇒ `APP_PASSWORD_MY_APP`. **2. Caddyfile route** — add a `handle` block above the catch-all: Open app (no auth): ```caddy handle /my-app/* { uri strip_prefix /my-app reverse_proxy 127.0.0.1:3001 } ``` Password-gated app: ```caddy handle /my-app/* { forward_auth 127.0.0.1:3000 { uri /auth/verify?app=my-app copy_headers Cookie @unauthorized status 401 handle_response @unauthorized { redir * /auth/login?app=my-app&next={http.request.uri.path} 302 } } uri strip_prefix /my-app reverse_proxy 127.0.0.1:3001 } ``` **3. Reload PM2 + Caddy:** ```bash pm2 restart ecosystem.config.js caddy reload --config ~/mini-apps/_registry/Caddyfile --adapter caddyfile ``` You do **not** need to touch Tailscale when adding an app. Caddy is the path router; Tailscale just sees one upstream (Caddy on `:8080`). **Verify:** ```bash curl -sI http://127.0.0.1:8080/my-app/ # expect 200 (open) or 302 (gated) ``` ## Removing a mini-app ```bash pm2 delete my-app ``` Then drop the Caddyfile `handle` block, the ecosystem entry, and any `APP_*_MY_APP` env vars from the auth-service block. Reload PM2 + Caddy. ## Slug convention for Hermes dashboards Hermes agent dashboards use the slug `hermes-` — `hermes-atlas`, `hermes-scout`, `hermes-vega`. Two reasons: 1. **Grouping** — every Hermes dashboard sorts together under one prefix. 2. **Collision avoidance** — a non-Hermes app may already own the bare agent name. If `/vega/` is already a web app on some port, the Vega _agent's_ Hermes dashboard still goes at `/hermes-vega/` (profile `vega`, its own port). Never mount a Hermes dashboard at a bare name that collides with an existing app slug. Assign the dashboards a contiguous port block (for example `9119`+, one per agent) and keep the mapping in your own registry. The env slug-to-password rule still applies: `hermes-atlas` ⇒ `APP_PASSWORD_HERMES_ATLAS`. ## Front-door outage: Caddy died (the #1 silent failure) Symptom: every app 502s at once, or `curl http://127.0.0.1:8080/health` returns `000` (connection refused), but `pm2 list` shows all the backend apps `online`. The router — Caddy — is dead and nothing restarted it. **Root cause is almost always that Caddy was started as a bare process, not under PM2.** A bare `caddy run …` (or one launched outside the ecosystem) has no supervisor, so when it crashes or the machine churns, it stays dead while every backend keeps running. The backends being healthy is what makes this confusing. **Recovery — bring the front door back, supervised this time:** ```bash export PM2_HOME=/Users//.pm2 # literal path, no ~ or $HOME under Hermes CADDY=$(command -v caddy || echo /opt/homebrew/bin/caddy) # 1. Find the config path the OLD process actually used — do NOT assume the skill's # default. The router dir may have been renamed (see next pitfall). ps aux | grep "caddy.*run" | grep -v grep # reveals --config if still running # 2. Validate before starting "$CADDY" validate --config --adapter caddyfile # 3. Start UNDER PM2 so it resurrects on crash pm2 start "$CADDY" --name caddy --interpreter none -- \ run --config --adapter caddyfile pm2 save # 4. Verify curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8080/health # expect 200 ``` After this, `pm2 list` should show a `caddy` entry. If it never had one, that was the bug — Caddy must always be a PM2 process. ## Reboot resurrection is a separate, latent outage `pm2 save` persists the process list, but it does NOTHING on reboot unless a launchd/ systemd boot hook exists. Check it explicitly — a missing hook is a time-bomb identical to the bare-Caddy outage, just triggered by a restart: ```bash launchctl list 2>/dev/null | grep -i pm2 || echo "NO pm2 boot hook — reboot resurrects nothing" ``` Fix (needs the user's sudo — surface it, don't try to run sudo yourself under Hermes): ```bash sudo env PATH=$PATH PM2_HOME=/Users//.pm2 pm2 startup launchd -u --hp /Users/ ``` ## The router dir may have been renamed (find it, never assume) The historical path was `~/openclaw-apps/` (pre-Hermes); the current one is `~/mini-apps/`. Older hosts may still carry the legacy directory name. Hardcoded paths in this skill or in old notes will `stat: No such file` even though the stack is alive. Locate the real one from ground truth rather than guessing: ```bash ps aux | grep "caddy.*run" | grep -v grep # --config reveals the registry dir find /Users/ -maxdepth 4 -name ecosystem.config.js 2>/dev/null | grep -v node_modules find /Users/ -maxdepth 5 -name Caddyfile 2>/dev/null | grep -v node_modules ``` The Caddyfile lives one level down, at `/router/Caddyfile` (current installer) or `/_registry/Caddyfile` (older hosts) — not directly in the router dir. Note also: under a Hermes tool environment, `$HOME` is rewritten to the profile home, so `~/mini-apps` and bare `ls`/`grep` may fail to see the real dir — use absolute `/Users//...` paths, and the Read/Grep tools (which resolve absolute paths) rather than shell `ls`/`cat` when the shell is sandboxed. ## Hermes dashboards behind the router > **Deep detail:** see `references/hermes-dashboard-rollout.md` for the full debugging > path — the Secure-cookie/HTTPS trap, the `--skip-build` web_dist crash-loop fix (a > FRESH host has no compiled `web_dist`; build once with > `cd ~/.hermes/hermes-agent/web && npm install && npm run build`), checking who owns > `:443` before exposure decisions, profile-vs-root DB selection, the SSH-hairpin > gotcha, matching a custom index page's design system when adding a dashboard card, and > the end-to-end verification sequence. Read it before any new `hermes-` rollout. **Slug convention:** mount Hermes dashboards at `/hermes-/` (e.g. `/hermes-atlas/`, `/hermes-orion/`) so the agent dashboard never collides with a same-named product app (`/nova/` = CFO app; `/hermes-nova/` = the agent). **Critical: the auth cookie is `Secure`, so the dashboard MUST be reached over HTTPS.** A plain-HTTP tailnet door (Tailscale `serve --http=PORT`) silently breaks login: `POST /auth/login` returns 303 + cookie, but the authed GET comes back 302 because the browser won't resend a Secure cookie over HTTP. If Caddy doesn't own a 443 funnel on the host, add a tailnet-only HTTPS door: ## Hermes dashboards behind the router A Hermes dashboard is a normal mini-app, with three extra requirements. **Fleet rollout addendum:** when exposing dashboards across multiple Hermes hosts, use `references/hermes-dashboard-fleet-rollout.md` for slug naming (`/hermes-$botname`), public Funnel/auth routing, Linux PM2/Caddy notes, and browser verification standards. **1. ecosystem.config.js** node-via-shell trick or PM2's interpreter override: ```js { name: "my-dashboard", script: "hermes", args: "dashboard --port 9120 --no-open --skip-build", interpreter: "none", cwd: "~/mini-apps", env: { PORT: 9120, HERMES_PROFILE: "my-profile", // omit if you want the root state DB }, }, ``` **2. Caddyfile** — add the prefix header so the SPA rewrites asset URLs correctly, but STILL keep `uri strip_prefix` — the FastAPI routes mount at root: ```caddy handle /my-dashboard/* { forward_auth 127.0.0.1:3000 { uri /auth/verify?app=my-dashboard copy_headers Cookie @unauthorized status 401 handle_response @unauthorized { redir * /auth/login?app=my-dashboard&next={http.request.uri.path} 302 } } uri strip_prefix /my-dashboard reverse_proxy 127.0.0.1:9120 { header_up Host {upstream_hostport} header_up X-Forwarded-Prefix /my-dashboard } } ``` `X-Forwarded-Prefix` is used by Hermes ONLY for HTML rewriting (asset paths and `__HERMES_BASE_PATH__`). API routes still mount at root, so `uri strip_prefix` is mandatory. **3a. The dashboard frontend must be built (`web_dist`) before `--skip-build` works.** On a fresh fleet machine the Hermes web UI is often unbuilt, and a dashboard launched with `--skip-build` will crash-loop with: `✗ --skip-build was passed but no web dist found at: …/hermes_cli/web_dist`. PM2 will show the process `errored` with a climbing restart count, and the port won't listen. Build it ONCE (takes ~1-2 min), then restart the PM2 process: ```bash cd ~/.hermes/hermes-agent/web && npm install --no-audit --no-fund && npm run build # build writes to../hermes_cli/web_dist/; then: PM2_HOME=/Users//.pm2 pm2 restart -dashboard PM2_HOME=/Users//.pm2 pm2 reset -dashboard # clear the crash-loop counter ``` Always check `ls ~/.hermes/hermes-agent/hermes_cli/web_dist/index.html` during pre-flight; some machines have it pre-built, others don't. **3b. Verify there are sessions to show — don't assume the profile name.** Some fleet agents run as root cron jobs against the root `~/.hermes/state.db`, not against `~/.hermes/profiles//state.db` (Scout, Orion, Ali use the root DB with NO `--profile` flag; Atlas/Vega/Nova use `--profile `). Pinning the wrong DB shows an empty dashboard. Quick triage: ```bash for db in $(find ~/.hermes -name state.db 2>/dev/null); do n=$(sqlite3 "$db" "SELECT COUNT(*) FROM sessions;" 2>/dev/null) echo " $n $db" done ``` Pin to whichever DB actually holds the sessions. **4. Build the dashboard web UI first, or `--skip-build` crash-loops.** The dashboard is served from a pre-built `web_dist/` directory. On a fresh Hermes install it may not exist yet, and PM2 will show the process `errored` with restart count climbing. Check `pm2 logs ` for: ``` ✗ --skip-build was passed but no web dist found at:.../hermes_cli/web_dist ``` Fix — build it once (takes ~1-2 min), then restart: ```bash cd ~/.hermes/hermes-agent/web && npm install && npm run build # emits ~/.hermes/hermes-agent/hermes_cli/web_dist/ pm2 restart && pm2 reset # reset clears the inflated restart counter ``` After that `--skip-build` is correct (fast start, no rebuild per boot). **5. Pin `HERMES_HOME` in env and pass the profile as a CLI arg, not env.** The reliable combo is `args: "--profile dashboard..."` (omit `--profile` for the root DB) plus `env: { HERMES_HOME: "/Users//.hermes", PATH: ":/opt/homebrew/bin:..." }`. Set `HERMES_HOME` to an ABSOLUTE path — under PM2 the rewritten `$HOME` otherwise points the dashboard at an empty DB. Use the absolute venv hermes binary as `script` (e.g. `/Users//.hermes/hermes-agent/venv/bin/hermes`), not the bare `hermes` name. **Verify session presence at the DB, NOT through the proxied API.** The Hermes dashboard's `/api/sessions` endpoint enforces its own token auth that only the in-browser SPA supplies — a plain `curl` (even with a valid mini-app auth cookie) gets `{"detail":"Unauthorized"}` or parses as 0 sessions. That is NOT evidence of an empty dashboard. Confirm data the honest way: ```bash sqlite3 /Users//.hermes/profiles//state.db "SELECT COUNT(*) FROM sessions;" # scout runs against the ROOT db: /Users//.hermes/state.db ``` The end-to-end proof for a Hermes dashboard is: backend port returns 200 on loopback + login round-trip serves the SPA shell (200, `Hermes Agent - Dashboard`) + the pinned DB has a non-zero session count. The SPA shell is ~700 bytes; data hydrates client-side after login. ## Public exposure via Tailscale Funnel Tailscale Serve is tailnet-only by default. To share an app publicly, add the funnel-enabled port to `~/mini-apps/_registry/tailscale-serve.json`: ```jsonc "AllowFunnel": { "${HOST}:443": true } ``` Then re-apply: `~/mini-apps/_registry/apply-tailscale-serve.sh`. **Funnel-allowed ports are exactly `{443, 8443, 10000}`.** Anything else fails silently or is rejected by Tailscale. **Security rule (non-negotiable):** a funnel'd port is fully public. Only password-gated or token-gated upstreams may live behind it. Put passwordless admin UIs on a separate **tailnet-only** port (e.g. `:8443`) by adding a `Web` entry _without_ a matching `AllowFunnel` entry. ### The Tailscale "Proxy" → loopback constraint The `Proxy` field in `tailscale-serve.json` (and `tailscale serve --bg http://…`) **only works with loopback backends** (`127.0.0.1:*`). Pointing a Serve/Funnel handler at a tailnet-IP backend (e.g. `100.x.y.z:20128`) returns HTTP 502 through the funnel and silently strips the Funnel off that port. **Fix pattern:** add a dedicated Caddy listener on a loopback port whose only job is to reverse-proxy to the tailnet-IP backend, then funnel that loopback listener. Example for an LLM proxy bound to a tailnet IP: ```caddy # In ~/mini-apps/_registry/Caddyfile, ABOVE the main:8080 block: :8090 { bind 127.0.0.1 reverse_proxy 100.x.y.z:20128 { header_up Host {upstream_hostport} } } ``` Then in `tailscale-serve.json`: ```jsonc "${HOST}:10000": { "Handlers": { "/": { "Proxy": "http://127.0.0.1:8090" } } }, "AllowFunnel": { "${HOST}:10000": true } ``` Use a dedicated listener (NOT a path under `:8080`) when the upstream is a SPA or Next.js app with absolute `/_next/static/...` asset paths — mounting it on a prefix inside `:8080` would 404 every asset. ## Hooks / webhooks (optional) Caddy can front a local agent gateway's webhook endpoint so external callers reach it over the same HTTPS door. **Hermes** runs its own webhook listener (`WEBHOOK_ENABLED=true`, `WEBHOOK_PORT`, default `8644`) and authenticates callers by **HMAC signature** using `WEBHOOK_SECRET`, not a bearer token. Caddy does not need to inject credentials — proxy the path and let the gateway verify the signature itself: ```caddyfile handle /hooks/* { reverse_proxy 127.0.0.1:8644 } ``` Point the sender at `https:///hooks/` and configure the same secret on both ends. Do **not** add the mini-app auth sidecar in front of this path — the caller is a machine and cannot complete a cookie login. A legacy agent runtime on the same host may instead expose a bearer-token `/hooks/*` endpoint, which needs token injection in the Caddy env block. Confirm which gateway owns the port before wiring it. ### Public exposure audit for hooks Do not infer public exposure from a process binding `0.0.0.0`. On these hosts there are two separate layers: - **Raw listener:** e.g. a local webhook process on `*:8644`; reachable on local/LAN/tailnet interfaces depending on firewall, but not necessarily public. - **Public front door:** Tailscale Funnel routes (`tailscale serve status`) usually point public HTTPS traffic at loopback Caddy, and Caddy may then expose selected paths such as `/hooks/*`. When asked "how is this public?" or before rolling out a webhook, verify all layers: 1. `lsof -nP -iTCP: -sTCP:LISTEN` — identify the raw listener and bind address. 2. `tailscale serve status` — identify which ports/paths are Funnel-enabled vs tailnet-only. 3. Inspect the active Caddyfile for `handle /hooks/*` or webhook-specific proxy blocks. 4. Check for PF/NAT redirects if the port appears reachable but is not in Funnel/Caddy. 5. Redact injected bearer tokens/secrets when reporting the route. A common safe conclusion: existing `/hooks/*` may be public via Funnel + Caddy bearer injection, while a separate Hermes webhook listener is only bound locally/tailnet and is not public until a Caddy/Tailscale route is added. ## The PM2 $HOME trap (CRITICAL when running tools inside Hermes) Hermes rewrites `$HOME` for tool execution to `~/.hermes/profiles//home/`. PM2 keys its socket off `$HOME`. Without `PM2_HOME` set, you talk to a **shadow** PM2 daemon that supervises nothing. **Always export this first:** ```bash # Real shell only — DO NOT use this under Hermes (both $HOME and ~ expand to the rewritten profile home): export PM2_HOME=$HOME/.pm2 ``` If you're calling PM2 from inside a Hermes tool environment, hardcode the absolute path with **no shell expansion** — neither `~` nor `$HOME` resolves to the user's real home under the rewritten environment: ```bash # Substitute your real username; do NOT use ~ or $HOME here. export PM2_HOME=/Users//.pm2 # macOS # export PM2_HOME=/home//.pm2 # Linux ``` Quick sanity check after exporting — this should print the real user's home, not a `.hermes/profiles/...` path: ```bash echo "$PM2_HOME" ``` Detect the trap: ```bash ps -ef | grep "PM2.*God" | grep -v grep ``` Two daemons with different `$HOME` paths = you spawned a shadow. Clean up: ```bash pm2 kill # kills the shadow export PM2_HOME=/Users//.pm2 # the real one — literal path, no ~ or $HOME pm2 list # now shows the actual fleet ``` Same trap applies to skill-asset paths: `~/mini-apps/...` resolves under the rewritten Hermes home. Hardcode the user's real path if you're scripting from inside a Hermes profile. ## Auth-service env reload trap The auth sidecar reads `APP_PASSWORD_*` / `APP_TITLE_*` / `APP_DESC_*` at **startup**. Changing a password in `ecosystem.config.js` and running `pm2 restart auth-service --update-env` does NOT pick up changes from the ecosystem file — `--update-env` only re-reads env from PM2's own state. To actually reload: ```bash export PM2_HOME=/Users//.pm2 # literal path — no ~ or $HOME under Hermes pm2 delete auth-service pm2 start ~/mini-apps/ecosystem.config.js --only auth-service ``` ## When Tailscale Serve goes sideways Any tool that runs `tailscale serve reset` on its own startup will wipe your config. Hermes' gateway does not manage Tailscale, so on a Hermes-only host suspect another service or a legacy agent runtime. Diagnosis: `tailscale serve status` shows the wrong routes (or nothing), or a funnel'd port disappeared. Recovery is one command: ```bash ~/mini-apps/_registry/apply-tailscale-serve.sh ``` The apply script is idempotent — safe to run any time. If a freshly added funnel route returns 502, work through this in order: 1. `tailscale funnel status` — confirm the funnel is still on for that port. If it's missing, you almost certainly pointed it at a non-loopback backend (see the loopback constraint above). 2. `lsof -nP -iTCP: -sTCP:LISTEN` — confirm the backend binds loopback. If it binds a tailnet IP only, use the Caddy-bridge pattern. 3. `curl -sI http://127.0.0.1:/` — direct loopback check. Connection refused = tailnet-IP-only backend. ## Login round-trip (for debugging an auth loop) **Always run a NEGATIVE auth test too** — a passing positive test doesn't prove the gate actually blocks. A correct gate returns 302 (redirect to login) for a wrong password: ```bash HOST=..ts.net JAR=/tmp/jar-bad.txt && rm -f $JAR curl -sk -o /dev/null -c $JAR -X POST "https://$HOST/auth/login" \ --data-urlencode "app=" --data-urlencode "password=WRONG" \ --data-urlencode "next=//" -H "Origin: https://$HOST" >/dev/null curl -sk -o /dev/null -w "wrong-pw authed GET => %{http_code}\n" -b $JAR "https://$HOST//" # expect 302 (denied). 200 here means the gate is broken/open. ``` A successful login returns 303/302 with a Set-Cookie; the authed GET then returns 200. ```bash JAR=/tmp/mini-app-cookies.txt && rm -f $JAR # 1. Login: 303 + Set-Cookie curl -si -c $JAR -X POST "http://127.0.0.1:8080/auth/login" \ --data-urlencode "app=my-app" \ --data-urlencode "password=the-password" \ --data-urlencode "next=/my-app/" \ -H "Origin: http://127.0.0.1:8080" -H "Host: 127.0.0.1:8080" | head -10 # 2. Authed GET — 200, not 302 curl -sI -b $JAR "http://127.0.0.1:8080/my-app/" | head -5 ``` ## End-to-end verification after any change ```bash # 1. PM2 supervises everything export PM2_HOME=~/.pm2 pm2 list # 2. Expected ports listen lsof -iTCP -sTCP:LISTEN -P -n | grep -E ":3000|:8080|" # 3. Caddy is on the latest config caddy reload --config ~/mini-apps/_registry/Caddyfile --adapter caddyfile # 4. Routes return expected codes for path in "" "auth/login?app=my-app" "my-app/" "health"; do code=$(curl -sk -o /dev/null -w "%{http_code}" \ "https://..ts.net/$path") echo " $code /$path" done # 5. Persist PM2 state across reboot pm2 save ``` Expected: - `/` → 200 (welcome page) - `/auth/login?app=…` → 200 - `//` → 302 (redirect to login when not authed) - `//` → 200 - `/health` → 200 - `/hooks/test` → 405 (POST-only) if hooks are enabled ## The auth cookie is `Secure` — gated apps REQUIRE an HTTPS door The auth sidecar sets `oc_auth_` with the `Secure` flag. A browser (and `curl` across hosts) will only send it back over **HTTPS**. Consequences: - A plain-HTTP tailnet door (e.g. `tailscale serve --http=4243`) will let you log in (303 + Set-Cookie) but the **authed GET still 302s** because the cookie never comes back. Gated apps are effectively unusable there. - **Loopback curl gives a false positive:** `curl -c jar … && curl -b jar …` over `http://127.0.0.1:8080` "works" because curl reuses the jar within the same call regardless of the Secure flag. Don't trust a loopback round-trip to prove auth — test over the real HTTPS tailnet/funnel URL. **Always front gated apps with an HTTPS door.** On a machine where Caddy owns `:443` (funnel), that's covered. On a machine where another service owns `:443`, add a **tailnet-only HTTPS door** on `:8443` pointed at Caddy's loopback `:8080`: ```bash tailscale serve --bg --https=8443 http://127.0.0.1:8080 # tailnet-only (no --funnel) — survives reboot natively. Verify: tailscale serve status # → https://..ts.net:8443 (tailnet only) ``` Then the app is reachable at `https://..ts.net:8443//`. **Correct end-to-end verification (run from a DIFFERENT host on the tailnet, over HTTPS):** ```bash BASE="https://..ts.net:8443"; JAR=/tmp/jar.txt; rm -f $JAR login=$(curl -sk -o /dev/null -w "%{http_code}" -c $JAR -X POST "$BASE/auth/login" \ --data-urlencode "app=" --data-urlencode "password=" \ --data-urlencode "next=//" -H "Origin: $BASE") authed=$(curl -sk -o /dev/null -w "%{http_code}" -b $JAR "$BASE//") echo "login=$login authed=$authed" # want 303 + 200; wrong password → authed 302 ``` Note: a box hitting its OWN tailnet hostname from inside an SSH session can hang (hairpin) — run the cross-host test from a different fleet machine instead. ## When NOT to take over the public `:443` funnel If another service already owns the public `:443` funnel — check `tailscale serve status` for an existing `:443 … proxy http://127.0.0.1:` entry — do NOT seize `:443` for Caddy without asking, because it disrupts whatever is already answering there. Default to a **tailnet-only HTTPS `:8443` door** (works on any device signed into the tailnet, not public). Add a public funnel later only on explicit request. This is the safe, reversible resting state. ## Pre-flight survey for a NEW machine (one SSH pass) Before installing anything on a fresh fleet host, gather all of this at once: ```bash # identity + prereqs whoami; echo $HOME; sw_vers -productVersion; uname -m ls /opt/homebrew/bin/brew /opt/homebrew/bin/caddy 2>/dev/null # caddy often MISSING which node pm2 hermes ls ~/.local/bin/hermes ~/.hermes/hermes-agent/venv/bin/hermes 2>/dev/null # sessions per DB (root vs profiles) — picks the right --profile flag for db in ~/.hermes/state.db ~/.hermes/profiles/*/state.db; do [ -f "$db" ] && echo "$(sqlite3 "$db" 'SELECT COUNT(*) FROM sessions') $db"; done ls ~/.hermes/hermes-agent/hermes_cli/web_dist/index.html 2>/dev/null # build needed? ls -d ~/mini-apps ~/openclaw-apps 2>/dev/null # existing router? # who owns:443 / what serve routes exist already? /opt/homebrew/bin/tailscale serve status ``` `tailscale` lives at `/opt/homebrew/bin/tailscale` on macOS (the GUI app's binary is also linked there); it's usually NOT on a non-interactive SSH PATH, so call it by full path or `export PATH=/opt/homebrew/bin:$PATH` first. Same for `brew`/`caddy`. nvm-based node lives at `~/.nvm/versions/node//bin`. ## Standing up the router on a bare machine `brew install caddy && npm install -g pm2`. There's no public installer for the auth-service on a bare box, so copy a known-good `auth-service/` from an existing fleet machine. **Use `scp`, not an SSH-pipe heredoc** — `tar czf - … | ssh host 'tar xzf -'` fails ("Unrecognized archive format") and base64-through-heredoc fails (the heredoc consumes stdin so the piped data never arrives). Reliable pattern: ```bash ssh src 'cd ~/mini-apps && tar czf - --exclude=node_modules auth-service' > /tmp/a.tgz scp /tmp/a.tgz dst:/tmp/a.tgz ssh dst 'cd ~/mini-apps && tar xzf /tmp/a.tgz && cd auth-service && npm install --no-audit --no-fund' ``` Then write `ecosystem.config.js` (fresh per-host `AUTH_SECRET` via `openssl rand -hex 32` — NEVER reuse between machines), the `Caddyfile`, and an index page, start all three under PM2 (auth-service, the dashboard, caddy), `pm2 save`, add the `:8443` serve, and do the HTTPS round-trip test. ## Run Caddy UNDER PM2 — not as a bare process If Caddy runs as a standalone process (not under PM2) and dies, nothing restarts it and the whole front door 502s while every backend stays healthy — a confusing outage where `pm2 list` looks fine but `https:///` is down. Diagnose: `ps aux | grep "caddy run"` shows nothing, `curl http://127.0.0.1:8080/health` returns 000. Fix and prevent: ```bash caddy validate --config --adapter caddyfile # validate first pm2 start /opt/homebrew/bin/caddy --name caddy --interpreter none -- \ run --config --adapter caddyfile pm2 save ``` ## Index-page cards: match the page's own design system When adding a dashboard card to an existing index page, READ the file first and reuse its existing card markup/classes (e.g. this fleet's pages use `.card` + `.pill`, not the older `.card-lock` span). Two failure modes seen: 1. **Orphaned card** — a naive "insert before ``" lands the card OUTSIDE the styled `
`/`.grid` container, so it renders as unstyled floating text. Insert INSIDE the `.grid` (e.g. right after `
`). 2. **Wrong markup** — copying the Mac Studio `.card-lock` pattern onto a page that only defines `.pill` yields an unstyled card. Use the target page's own classes. After editing, the served HTML is authoritative; a stale browser view is just cache (hard-refresh ⌘⇧R). Verify with `curl -sk "$BASE/" | grep -n hermes-` and confirm the card line sits between `
` and `
`. ## Reboot-resurrect needs sudo (flag it, don't skip it) `pm2 save` persists the process list, but resurrection on reboot needs a launchd entry from `sudo pm2 startup` — which can't run unattended. Check `launchctl list | grep -i pm2`; if absent, surface the exact command for the user to run (it prints from `pm2 startup launchd -u --hp /Users/`) rather than silently leaving the host unable to recover from a reboot. Tailscale `serve --bg` config persists across reboots natively, so only PM2 needs this. 1. **Editing the Caddyfile in `/etc/...` or wherever you found Caddy on the system.** The router uses `~/mini-apps/_registry/Caddyfile` and runs Caddy under PM2 with `--config` pointing at that file. Edit there, reload from there. 2. **Forgetting `uri strip_prefix /` on a Hermes dashboard.** Hermes receives `X-Forwarded-Prefix` but only uses it for HTML rewriting. The API routes mount at root and 404 without strip-prefix. 3. **`pm2 restart auth-service --update-env` after editing `ecosystem.config.js`.** `--update-env` doesn't re-read the file. Use `pm2 delete` + `pm2 start --only` to actually reload. 4. **Pointing Tailscale Serve/Funnel at a non-loopback backend.** Returns 502 and strips Funnel off the port silently. Use a Caddy loopback bridge. 5. **Putting passwordless admin UIs on a funnel'd port.** A funnel'd port is fully public. Move them to `:8443` (tailnet-only) by adding a `Web` entry without an `AllowFunnel` entry. 6. **Letting a second process manage Tailscale Serve.** Anything that runs `tailscale serve reset` on startup wipes these routes on every restart. Hermes' gateway has no Tailscale integration, so on a Hermes-only host look for another service or a legacy agent runtime; disable its serve management, then re-apply with `apply-tailscale-serve.sh`. 7. **Calling PM2 from inside a Hermes tool environment without exporting `PM2_HOME`.** You talk to a shadow daemon that supervises nothing. Export first, every time. 8. **Adding a port outside `{443, 8443, 10000}` to `AllowFunnel`.** Tailscale rejects it silently. Stick to those three for public exposure. 9. **`pm2 save` without `PM2_HOME` exported.** You save the wrong process list to the wrong dump file, and reboot resurrects nothing. 10. **Slugs with uppercase, underscores, or `>32` chars.** The auth sidecar rejects them at `/auth/verify` and `/auth/login` with 400. Pattern is `^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$`. 11. **Serving a password-protected app over plain HTTP.** The session cookie is `Secure`, so browsers and `curl` refuse to send it back over `http://`. Symptom: login POST returns 303 (looks like success) but every subsequent GET bounces back to the login page because the cookie never returns. ALWAYS use an HTTPS door: `tailscale serve --bg --https=8443 http://127.0.0.1:8080` for tailnet-only, or the `:443` HTTPS funnel. Beware a pre-existing plain-HTTP `:4243`-style serve listener — it's the trap. When testing cross-node, use the `https://` URL with `curl -sk`; a loopback test on the box can falsely "pass" because curl reuses the jar within one invocation regardless of Secure. 12. **`pm2 startup` needs sudo and bakes in the wrong `--hp` under Hermes.** Reboot resurrection requires a launchd entry (`launchctl list | grep pm2`). Generating it needs an interactive sudo, so it CANNOT be done unattended — hand the user the exact command. Generate it with a clean HOME or the rewritten profile `$HOME` leaks into `--hp`: `HOME=/Users/ PM2_HOME=/Users//.pm2 pm2 startup launchd -u --hp /Users/`. The `--hp` must match where `dump.pm2` actually lives. The `pm2 save` dump and `tailscale serve --bg` config both persist across reboot on their own; only the launchd entry needs the one-time sudo. 13. **Streaming a tarball over `ssh... 'bash -s' < --name caddy --interpreter none -- run --config...`). Diagnosis: backends `online` in `pm2 list` but `curl http://127.0.0.1:8080/health` is refused/000 ⇒ Caddy itself is down. 15. **Starting a Hermes dashboard with `--skip-build` on a fresh host that never built the frontend.** The process crash-loops `errored` with `✗ --skip-build was passed but no web dist found at: …/hermes_cli/web_dist`. Build it once: `cd ~/.hermes/hermes-agent/web && npm install && npm run build`, then restart. Pre-flight: `ls ~/.hermes/hermes-agent/hermes_cli/web_dist/index.html || echo "build web_dist first"`. 16. **Reaching a Hermes dashboard over a plain-HTTP tailnet door.** The auth cookie is `Secure`, so it's never resent over `http://` — login returns 303 but the authed GET stays 302 (looks like a broken gate). Serve over HTTPS: a 443 funnel if Caddy owns it, else a tailnet-only HTTPS door `tailscale serve --bg --https=8443 http://127.0.0.1:8080`. Always run the login round-trip over the `https://` URL — an HTTP test gives a false negative. 17. **Pasting generic card markup into a host's custom index page.** Index pages differ per host: an inline Caddyfile `respond` block (Mac Studio) vs a file_server `_registry/index.html` with a bespoke theme. READ the existing index first and reuse ITS card/pill classes; insert the new card INSIDE the existing grid/container, never after `` (an orphan outside the styled wrapper renders as raw, unstyled floating text — a user-visible "looks funny" defect). Back up before editing; verify with a cache-busting browser reload. ## Veri …(truncated)