Extension E2e Test

Extension E2E Test

End-to-end validation of a pi extension in an isolated herdr workspace: load the extension alone, drive pi like a user (send messages, switch models), and assert on the footer status line.

🔴 Requires herdr: works only inside herdr (HERDR_ENV=1). Check test "${HERDR_ENV:-}" = 1; if unset, stop — the CLI talks to the running instance over a unix socket and has no effect outside.

The two command surfaces

herdr exposes the same terminal at two levels; pick the right one.

• agent surface — herdr has recognized which coding agent occupies the pane (pi, claude, codex, …) and drives it through a lifecycle state machine: idle (ready), working, blocked (approval/question UI), done, unknown (unclassifiable). Commands: agent prompt / wait / get / list / read / send-keys.
• pane surface — the terminal as a byte stream. Commands: pane run / send-text / send-keys / read / wait-output.

Prefer the agent surface. It waits on real state transitions instead of grepping the viewport (which false-matches text already on screen), and agent prompt --wait is an atomic submit-and-wait whose stall guard returns agent_prompt_stalled if pi shows no lifecycle change within 5s. That is the structural cure for false completion — the message was sent but pi never ran it, so a bare "wait idle" matches the still-idle pane. The agent surface also sees blocked approvals the pane surface is blind to.

The one prerequisite — and the trap. Detection works only while herdr can read pi's state. For pi that signal comes from a pi extension, ~/.pi/agent/extensions/herdr-agent-state.ts, which reports state to herdr (this is why detection is reliable: screen_detection_skipped: true, no scraping). The isolation this skill uses (--no-extensions) disables that extension and drops pi to unknown. The fix is to load it back explicitly — the second -e in step 2. If a target ever reads unknown, drop to the pane surface (REFERENCE.md → Fallback).

Prerequisites

• herdr on PATH and running inside it (HERDR_ENV=1)
• pi available (compiled or via npx)
• the extension source path known
• ~/.pi/agent/extensions/herdr-agent-state.ts present (herdr integration install pi)

Workflow

1. Create an isolated workspace

EXT_PATH="$(pwd)/packages/<pkg>/extensions/<file>.ts"
WS_JSON=$(herdr workspace create --cwd "$(pwd)" --label "pi-e2e-<ext>" --no-focus)
ROOT_PANE=$(echo "$WS_JSON" | python3 -c 'import sys,json; print(json.load(sys.stdin)["result"]["root_pane"]["pane_id"])')
WID=$(echo "$WS_JSON" | python3 -c 'import sys,json; print(json.load(sys.stdin)["result"]["workspace"]["workspace_id"])')

--no-focus keeps the user's focus. Done when both ROOT_PANE and WID are parsed from the JSON — always parse fresh, since IDs compact when workspaces close.

2. Start pi in isolation

# -ne: no auto-discovered extensions · -ns: no skills · -e: load only what's named
herdr pane run "$ROOT_PANE" \
  "pi --no-extensions --no-skills -e '$EXT_PATH' -e '$HOME/.pi/agent/extensions/herdr-agent-state.ts'"

The second -e (herdr-agent-state.ts) keeps detection alive: -e is repeatable and -ne only disables auto-discovery, so explicit -e paths still load — fully isolated, yet herdr still sees pi. Log through pane read --source visible; redirecting pi's stdout (> file, 2>&1 | tee) breaks the PTY and TUI rendering. Done when the command is sent (readiness is confirmed next).

3. Wait for readiness and confirm detection

herdr agent wait "$ROOT_PANE" --until idle --timeout 20000
herdr agent get "$ROOT_PANE"
herdr pane read "$ROOT_PANE" --source visible --lines 50

Done when agent get reports pi (not unknown) and the empty footer is visible (e.g. inference-speed shows --.-T/s FT -.-s). If unknown, re-check the second -e in step 2, else use the pane fallback (REFERENCE.md → Fallback).

4. Send a message and verify the footer

# Fast models: 30000. Reasoning models (high/xhigh thinking): 120000+.
herdr agent prompt "$ROOT_PANE" "<test prompt>" --wait --timeout 120000
herdr pane read "$ROOT_PANE" --source visible --lines 50

agent prompt --wait waits for the first settled state observed after submission, so its stall guard rules out false completion. Done when the footer shows populated values (e.g. inference-speed 16.0T/s FT1.6s), not the empty placeholder. Repeat for a second turn to verify cross-turn state.

A cheap fast model for routine testing: opencode-go/deepseek-v4-flash. For multi-model, slash-command, and multi-turn variants see REFERENCE.md.

5. Clean up

herdr workspace close "$WID"

Terminates pi and everything in the workspace. send-keys rejects control characters (no C-d), so cleanup is always workspace close. Done when close returns ok.

Command reference

agent wait states: idle/working/blocked/done/unknown; without --until it matches idle/done/blocked. These waits are indefinite without --timeout — always pass one. Exit code 0 = matched/settled, non-zero = timeout; there is no matched field to inspect.

Footer extraction

pane read --source visible --lines N returns the viewport as plain text with no ANSI escapes (the default text format strips styling), so grep it directly:

herdr pane read "$ROOT_PANE" --source visible --lines 50 | grep "<status-key>"

For ctx.ui.setStatus(key, text), grep the key to isolate the footer line — the last non-empty line, with the model/context line just above it. To assert populated-vs-empty, match a pattern only the populated state has: inference-speed's empty --.-T/s has no digits, so grep -E '[0-9]+\.[0-9]T/s' matches only a populated footer.

Always --source visible and always --lines N for TUI apps: recent/recent-unwrapped read scrollback (empty for full-screen TUIs) and the default line count is 1. Only --format ansi carries escapes — strip before grepping: sed -E "s/$(printf '\033')\[[0-9;]*m//g".

Disclosed reference

Branches and troubleshooting live in REFERENCE.md: the pane-surface fallback (when detection is unknown), multi-model testing, slash-command testing, multi-turn verification, startup options, log analysis, and a symptom→fix table.