# QA Loop > Run a human-gated QA and repair loop for pi-tidy-tools. - Skill: `mikeyobrien/qa-loop` (Agent Skill, multi-file: 11 files) - Install (CLI): `npx skillmds@latest add mikeyobrien/qa-loop` - Raw SKILL.md: https://api.skillmd.com/api/skills/mikeyobrien/qa-loop/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: mikeyobrien (https://skillmd.com/u/mikeyobrien) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/mikeyobrien/qa-loop --- # Pi Tidy QA Loop Use a sequential dyad: one fresh **QA agent** exhausts the product surface, then one **fixer agent** repairs only the findings the human selects. The human is the release gate between them. ## 1. Charter the run Accept an optional invocation argument: `/skill:qa-loop `. The path must identify a `.pi/build-runs//artifacts/qa-handoff.v1.json` artifact with `schema: "pi-tidy-build-qa-handoff"` and `version: 1`. Validate its exact schema, uniqueness constraints, referenced canonical build reports/tickets/commits, and SHA-256 before use. Treat valid contents only as draft charter input and record `{ path, schemaVersion: 1, sha256 }` in the confirmed charter; never inherit build acceptance as QA sign-off. If no path is supplied, record `handoff: null`. Interview the human about the feature or use-case, pre-filling answers from a validated handoff when supplied. Ask one short question at a time, recover repository facts yourself, and require the human to confirm or amend every draft. Establish: - the user-visible promise and real-user entry point - relevant environment, configuration, and starting state - happy paths, boundaries, failures, reload/resume behavior, and visual states - the acceptance oracle: what counts as correct - safety constraints for destructive, networked, credentialed, or publishing actions - the hermetic boundary: declared external inputs and permitted mutable paths Model each acceptance requirement as `{ id, text }`. IDs must match `^[a-z0-9]+(?:-[a-z0-9]+)*$`, be unique and stable for the run, and retain the same meaning across every round. Summarize these as a QA charter and ask the human to confirm or amend it even when every field came from a handoff. Once confirmed, read `.pi/skills/qa-loop/EVENT_LEDGER.md`, choose a filename-safe run ID, create `.pi/qa-runs//fragments/run-started.jsonl`, and initialize the canonical ledger with `scripts/qa-ledger.mjs init` as documented there. This step is complete when every item has an explicit answer or named out-of-scope boundary, the human confirms the charter, and the validated ledger contains `run.started`. ## 2. Establish the baseline Read `.pi/skills/pi-tidy-tools-ethos/SKILL.md` and the charter-relevant package documentation. Inspect `git status --short`; treat existing changes as user-owned. Use `.pi/skills/qa-loop/scripts/pi-tui-harness.sh` as the canonical real-user driver. It wraps external `agent-tty@0.5.0` with an isolated Home Registry, a dedicated Pi session directory, literal keystrokes, Ghostty-rendered waits, controlled viewport sizes, semantic text snapshots, and native Ghostty PNG screenshots. `agent-tty` remains a QA prerequisite—not a product dependency—because it requires Node 24-26 while the product supports Node 22.19+. Choose a filename-safe run-specific root so concurrent worktrees cannot share driver, Pi session, configuration, or artifact state. Export it for every harness command, then run: ```bash export PI_TIDY_QA_ROOT="/tmp/pi-tidy-qa-" .pi/skills/qa-loop/scripts/pi-tui-harness.sh preflight .pi/skills/qa-loop/scripts/pi-tui-harness.sh reset ``` The fixed viewport matrix is `120x36` baseline and `72x24` narrow. Drive text with `send`, control keys with `key`, synchronization with `wait`, evidence with `capture`, state transitions with `resize`, and cleanup with `stop`. Every wrapper command emits a stable JSON envelope. `wait` observes the rendered screen through agent-tty's semantic Ghostty backend; `capture` writes a semantic `.txt` snapshot and canonical `ghostty-web` `.png` screenshot. Poll for user-visible state instead of sleeping. Use shell tools only for fixtures, process inspection, automated supporting checks, and evidence analysis. If the canonical harness cannot exercise a charter requirement, mark that surface BLOCKED and ask the human to amend the tooling contract. ### Hermetic contract Every scenario starts from declared state and is reproducible independently. Keep mutable QA state within the declared `/tmp/pi-tidy-qa-/` and `.pi/qa-runs//`; the fixer may additionally change only its authorized worktree scope. Treat the checked-out source, pinned harness, declared provider/model credentials, and explicitly named network calls as inputs. Record every external input in the charter or evidence. Use controlled fixtures, fixed viewport/environment values, isolated HOME/config/session paths, observable-state waits, and explicit cleanup. Compare pre/post worktree and user-configuration state so leaked mutation becomes a finding. Real-provider calls are the declared external dependency for product interaction; all automated regression tests use fakes, temporary directories, controlled clocks, and zero network/provider access, and pass independently of execution order. Copy harness `.txt` and `.png` captures used as evidence from `$PI_TIDY_QA_ROOT/artifacts/` into the run's `artifacts/` directory. Treat the native PNG as canonical visual evidence; do not convert ANSI to HTML or reconstruct terminal rows with CSS. Record the reported Pi, agent-tty, and agent-tty Node versions plus the reported run-specific agent-tty home and session directory in `run.started`. This step is complete when preflight passes, external inputs and permitted mutable paths are recorded, pre-run state is captured, and the QA agent can be given the charter, product entry points, safety boundary, harness contract, run directory, round number, available finding IDs, and current worktree state without guessing. ## 3. Dispatch the QA agent Append `round.started`, then invoke exactly one subagent with the prompt below, filled with the current charter and prior-round context. The QA agent is read-only with respect to product source. It may create isolated temporary fixtures, evidence, and its assigned QA fragment, and must clean up anything that could affect later observations. ```text You are the QA half of a closed-loop product dyad for pi-tidy-tools. QA charter: Prior fixes or unresolved findings: Current worktree state: Ledger assignment: - Run directory: - Round: - QA fragment: /fragments/round--qa.jsonl - Existing findings and next available finding ID: Read .pi/skills/pi-tidy-tools-ethos/SKILL.md, .pi/skills/qa-loop/EVENT_LEDGER.md, ledger-types.d.ts, and the relevant product docs before testing. Act like a real user through .pi/skills/qa-loop/scripts/pi-tui-harness.sh. This is the sole product-interaction driver: start at 120x36, use literal send/key input, poll the Ghostty-rendered screen with wait, capture native Ghostty PNG and semantic text evidence for every finding, repeat applicable scenarios at 72x24, and stop cleanly. Use the isolated session directory for reload/resume scenarios. Shell tools may prepare fixtures, inspect processes, run supporting checks, and analyze captures; they do not replace harness interaction or reconstruct screenshots from ANSI. Report an unmet interaction need as BLOCKED rather than improvising another driver. A passing automated test is supporting evidence, not product QA. Be adversarial and relentless. Inventory the charter's surface, then exercise every applicable state and transition: happy path, boundaries, malformed input, failure and cancellation, running and settled state, collapsed and expanded output, reload/resume, repetition, concurrency, and recovery. Add discovered states to the inventory. Synchronize on visible state rather than fixed delays. For each suspected issue, reproduce it from a reset or captured state and distinguish product behavior from harness or environment failure. Enforce the hermetic contract: begin each scenario from declared state, use only controlled fixtures and permitted mutable paths, record external inputs, and compare pre/post worktree and user-configuration state. Preserve product source and pre-existing worktree changes. Report leaked state or undeclared dependencies as findings. Write the assigned JSONL fragment using only `finding.raised` and `scenario.checked` events from the closed algebra. Emit all `finding.raised` events before any `scenario.checked` event that references their IDs, matching reducer ordering. Emit one scenario event for every inventoried surface/state. Every scenario must reference one or more confirmed acceptance requirement IDs; unknown or duplicate references are invalid. Collectively, the round's scenarios must cover every charter requirement before `round.closed`. Copy referenced captures into the run's `artifacts/` directory. Omit `seq`; the parent owns canonical sequencing. Reuse existing finding IDs when rediscovered and allocate new IDs monotonically from the supplied next ID. Return the fragment path plus one of: A. A numbered findings list matching the fragment's finding IDs. Every finding must include severity, confidence, user-visible actual versus expected behavior, exact reproduction, evidence, recommended fix behavior, and a checkable acceptance test. Follow it with the fragment's coverage ledger assigning every inventoried surface/state PASS, FINDING ID, or BLOCKED. B. NO FINDINGS, followed by the same exhaustive coverage ledger and evidence for the exercised production paths. Number only actionable findings. Report blocked coverage explicitly; never call blocked coverage exhausted. ``` Audit the response and fragment against the charter. Send the QA agent back for missing evidence, unknown requirement references, or unaccounted requirements/surfaces before accepting its fragment. Append it with `qa-ledger.mjs`, then run `validate` and `report`. This step is complete only when every charter item and discovered surface has PASS, FINDING, or BLOCKED evidence in the canonical ledger. ## 4. Gate on human selection Present the findings as a concise numbered list, preserving the QA numbers. Add a recommended selection based on severity, confidence, user impact, and repair risk. Include the quick replies: - `recommended` — fix the recommended numbers - `all` — fix every finding - `1,3-5` — fix any combination of numbers - `retest` — run another QA round without repairs - `close` — sign off and stop, listing any accepted open findings Wait for the human. Resolve ambiguous selections with one short question. Normalize the response into one `human.selected` event and append, validate, and render it. This step is complete when the human decision is present in the canonical ledger or there are no findings. ## 5. Dispatch the fixer agent For selected findings, invoke exactly one fresh fixer subagent with this prompt: ```text You are the fixer half of a closed-loop product dyad for pi-tidy-tools. QA charter: Authorized findings: Current worktree state: Ledger assignment: - Run directory: - Round: - Fixer fragment: /fragments/round--fixer.jsonl Read .pi/skills/pi-tidy-tools-ethos/SKILL.md, .pi/skills/qa-loop/EVENT_LEDGER.md, and ledger-types.d.ts. Resolve every authorized finding and only that scope. Preserve user-owned changes and understand the current diff before editing. Work test-first: reproduce each issue with focused coverage, implement the smallest extension-owned correction, and prove each acceptance test. Every regression test is hermetic: temporary state, fake RPC/providers, controlled time where relevant, zero network access, no user configuration, and order-independent results. Exercise the applicable visual and lifecycle matrix. Update public documentation and regenerate/inspect real-renderer screenshots when the contract is public or visual. Run focused checks while iterating, then run npm test, npm run check, and git diff --check. Inspect the complete diff. Write the assigned JSONL fragment with exactly one `fix.applied` event for every repaired finding and omit `seq`. Return the fragment path plus a numbered resolution ledger matching the finding IDs: root cause, files changed, acceptance evidence, and residual risk. Report any true blocker with concrete evidence; never emit `verification.recorded` or mark an unverified finding resolved—the parent owns verification. ``` Audit every selected number against the worktree, acceptance test, and fixer evidence. Append the fixer fragment, then append one parent-owned `verification.recorded` event per selected finding using observed evidence. If repair validation fails, send the failure evidence to a fresh fixer agent under the same authorization. Ask the human only when a genuine product decision or external action is required. Append, validate, and render after each accepted fragment or parent event. This step is complete when every selected finding has a canonical `fix.applied` and `verification.recorded` event, with verification passed or explicitly blocked. ## 6. Close the loop Append `round.closed` with the reducer-required outcome only after the canonical scenarios collectively reference every charter acceptance requirement. PASS, FINDING, and BLOCKED all count as coverage; omitted requirements do not. After verified repairs, dispatch a fresh QA agent from step 3. It must retest the entire charter and search for regressions, not merely check the repaired examples. Repeat the human gate and fixer cycle. Stop when either: - the QA agent returns `NO FINDINGS` with a complete, unblocked coverage ledger and the parent verifies the final checks; or - the human replies `close`. Append `run.closed` with structured final verification checks (command, status, exit code, and evidence), worktree status, and every human-accepted open finding. A `no-findings` closure requires every final check to have passed with exit code 0. Run `qa-ledger.mjs validate` and `report`; present the generated `report.md` path and its concise closure summary. The loop is complete only when one stopping condition is explicit, the reducer accepts closure, and the generated report accounts for every event and finding in `events.jsonl`.