Senpi QA

senpi QA

QA the senpi coding agent (packages/{ai,agent,coding-agent,tui}) by driving the REAL CLI — not by reading code or trusting unit tests. Each channel runs the agent from source via tsx in an isolated sandbox and asserts observable behavior, so a passing run is evidence the user-facing surface actually works.

Every helper script ships a --self-test (or --self-check) that asserts its scenario against this machine. The scripts are therefore both the QA tools and their own regression checks.

Golden rules (read before running anything)

• Isolation is mandatory. Everything spawns the CLI with SENPI_CODING_AGENT_DIR / SENPI_CODING_AGENT_SESSION_DIR pointed at a temp sandbox and PI_OFFLINE=1. QA must never write into the real ~/.senpi. scripts/lib/common.mjs does this for you — use it.
• Never read or modify the real credentials. ~/.senpi/agent/auth.json is the user's real key store. Every script snapshots its sha256 and asserts it is unchanged at the end. If you script a run by hand, do the same (guardRealAuth() in common.mjs).
• Deterministic loop = mock loop. To exercise the agent loop without real tokens, use Channel 3 (a local fake model server). Real-provider runs are for final smoke only and must use the user's existing auth, never a new key.
• No src/ edits from this skill. It verifies; it does not fix. If QA finds a bug, report it with the captured evidence and let a follow-up change fix it.
• The captured artifact IS the evidence. Write it under local-ignore/qa-evidence/<YYYYMMDD>-<slug>/. No artifact == the QA did not happen. local-ignore/ is gitignored — never commit evidence.

Setup (once)

node scripts/devenv-setup.mjs        # installs skill deps (node-pty), wires .env.local + .claude/skills
node .agents/skills/senpi-qa/scripts/lib/common.mjs --self-check   # confirm the harness

common.mjs --self-check confirms the repo resolves, a sandbox is created and auto-removed, a free port is allocatable, and the real auth file is untouched.

Router: match QA to your change

When in doubt, run the channel closest to your change AND Channel 3 (mock loop): the mock loop is the cheapest end-to-end proof that the agent still completes a turn.

Channels

All commands are run from the repo root.

Channel 1 — Remote RPC (scripts/rpc-drive.mjs)

Drives --mode rpc (JSON lines over stdio). get_state round-trips with no API call; --prompt drives a real turn and captures the event stream.

node .agents/skills/senpi-qa/scripts/rpc-drive.mjs --self-test
node .agents/skills/senpi-qa/scripts/rpc-drive.mjs --state
node .agents/skills/senpi-qa/scripts/rpc-drive.mjs --prompt "say PONG" --provider mock --model mock-model --evidence rpc-pong

Channel 2 — TUI smoke (scripts/tui-smoke.mjs)

Boots the interactive TUI in a real pseudo-terminal, confirms it renders and a keystroke reaches the composer, then tears it down. Uses node-pty (ConPTY on Windows — no WSL) and falls back to tmux on POSIX.

node .agents/skills/senpi-qa/scripts/tui-smoke.mjs --self-test
node .agents/skills/senpi-qa/scripts/tui-smoke.mjs --self-test --driver tmux --evidence tui

TUI smoke proves boot/render/input, not fine-grained output. For behavioral assertions use Channel 1 or 3.

Channel 3 — Mock loop (scripts/mock-loop.mjs)

Starts a local fake model server, registers it via a baseUrl override in an isolated models.json, and drives a REAL turn — deterministic, zero tokens. Covers all three wire formats senpi uses, so baseUrl override is QA'd for both OpenAI and Anthropic (pick with --api; default openai-completions):

--self-test (no --api) round-trips all three and exercises the three error/retry scenarios below. --with-tool proves the full loop (model → bash tool → final text). --with-mcp-tool registers a sandbox extension that proxies mcp_fx_tool_<n> to the local MCP stdio fixture, then asserts the fixture call log exists and the model's second request contains the fixture result. The loop is hermetic: provider key env vars are stripped so only the inline mock key is ever used.

node .agents/skills/senpi-qa/scripts/mock-loop.mjs --self-test
node .agents/skills/senpi-qa/scripts/mock-loop.mjs --self-test --api anthropic-messages
node .agents/skills/senpi-qa/scripts/mock-loop.mjs --with-tool --api openai-responses
node .agents/skills/senpi-qa/scripts/mock-loop.mjs --with-mcp-tool mcp_fx_tool_1 --tool-args '{"value":"ok"}'
node .agents/skills/senpi-qa/scripts/mock-loop.mjs --with-eval-hard-limit --evidence eval-hard-limit
node .agents/skills/senpi-qa/scripts/mock-loop.mjs --scenario transient-recover
node .agents/skills/senpi-qa/scripts/mock-loop.mjs --scenario budget-exhaust
node .agents/skills/senpi-qa/scripts/mock-loop.mjs --scenario long-retry-after
node .agents/skills/senpi-qa/scripts/mock-loop.mjs --run "summarize this repo" --evidence mock-summary

Channel 4 — CLI smoke (scripts/cli-smoke.mjs)

Fast, no model: --help, --version, offline --list-models, unknown-flag handling.

node .agents/skills/senpi-qa/scripts/cli-smoke.mjs --self-test

Channel 5 — Persistent terminal / PTY (scripts/pty-drive.mjs)

Drives the real @earendil-works/pi-pty runtime that backs the built-in terminal tools (bash / bash_output / bash_input / bash_resize / kill_bash) through the canonical scenarios: background command + monitor-style line watch and peek, stdin steering, screen snapshot + resize reflow, and registry teardown with no orphans. Uses native PTY when a host prebuild is present, else the pipe fallback (--force-pipe forces it).

node .agents/skills/senpi-qa/scripts/pty-drive.mjs --self-test --evidence terminal
node .agents/skills/senpi-qa/scripts/pty-drive.mjs --self-test --force-pipe

Scripts index (each is its own regression test)

Run the whole suite:

for s in lib/common.mjs:--self-check lib/fake-model-server.mjs:--self-test \
         rpc-drive.mjs:--self-test mock-loop.mjs:--self-test \
         tui-smoke.mjs:--self-test cli-smoke.mjs:--self-test; do
  node ".agents/skills/senpi-qa/scripts/${s%%:*}" "${s##*:}" || echo "FAILED: $s"
done

Capturing evidence

ev="local-ignore/qa-evidence/$(date +%Y%m%d)-senpi-qa-<slug>"; mkdir -p "$ev"
node .agents/skills/senpi-qa/scripts/mock-loop.mjs --self-test | tee "$ev/mock-loop.txt"
node .agents/skills/senpi-qa/scripts/rpc-drive.mjs --prompt "say PONG" \
  --provider mock --model mock-model --evidence senpi-qa-<slug>

Most channels accept --evidence <slug> and write artifacts to local-ignore/qa-evidence/<date>-<slug>/ themselves.

References

• references/rpc-protocol.md — RPC command/response catalog, turn completion, examples
• references/tui-driving.md — node-pty vs tmux, keybindings files, fragility, isolation
• references/mock-loop.md — fake server, custom-provider models.json shape, in-process faux alternative
• references/credential-injection.md — per-harness credential injection + masking
• references/env-vars.md — provider keys + isolation env vars