Orch QA

QA Engineer (orchestrator)

A framework-agnostic senior QA/QC engineer that evaluates existing codebases from multiple quality perspectives. Runs existing tests, triages failures, identifies missing test coverage, and optionally writes missing tests.

The primary deliverable is an evidence package — a timestamped directory containing proof artifacts (test output, failure details, gap proof files, screenshots) with REPORT.md as the index/navigator.

Role definition lives in the subagent

The QA engineer judgment — lens definitions, failure triage rubric, severity calibration, test-writing rules, evidence shape — lives in the @agent-orch-qa:orch-qa subagent. This skill is the orchestrator: it runs the long-lived shell-heavy phases (recon, monorepo selection, stack confirmation, preflight safety, test execution with recording, parallel team spawn for gap analysis) and delegates judgment calls to the subagent.

Use this skill when the workflow needs all of: project recon + execution + gap analysis + report. Use @agent-orch-qa:orch-qa directly when another agent has the project context and only needs the judgment call (failure triage on a specific failure, gap classification under a single lens, test-writing for a specific gap).

Locating your reference files

Every references/*.md link below is relative to this skill's own install location, not to the project you're running against — your working directory throughout this workflow is the target project (whatever --scope covers), so a bare relative path will not resolve. Resolve it once, in one Bash call, before the first reference read:

SKILL_HOME=$(ls -d ~/.claude/plugins/cache/hideki-plugins/orch-qa/*/skills/orch-qa 2>/dev/null | sort -V | tail -1)
echo "$SKILL_HOME"

Re-derive $SKILL_HOME in the same Bash call as any later reference read, or a fresh one — shell state (including this variable) does not persist between separate Bash tool calls. If it comes back empty, the skill isn't installed the normal way; proceed on the inline summaries in this file and note in REPORT.md that a given reference wasn't reachable rather than guessing at its content.

Arguments

Parse these from the user's invocation. All are optional with defaults:

Workflow Phases

Feedback Check

If feedback/log.md exists and has 5 or more entries, read the last 10 entries. If a pattern is apparent (same issue in 3+ entries, or average rating below 3):

• Tell the user: "Recurring feedback detected: [brief pattern]. Consider running /skill-improve --skill orch-qa."
• Continue with normal execution.

Phase 1: Reconnaissance

Understand the project before touching anything.

1. Read project root: README.md, package.json, pyproject.toml, go.mod, Cargo.toml, *.sln, etc.
2. Identify source directories vs test directories
3. Count source files vs test files — compute a rough test-to-source ratio
4. Map the directory tree (top 3 levels)
5. If --scope=changed, run git diff --name-only against the base branch to scope files
6. If --scope is a path/glob, restrict all subsequent phases to matching files

Monorepo handling:

• Detect workspace definitions (workspaces in package.json, pnpm-workspace.yaml, Cargo workspace, Go workspace)
• If monorepo detected, ask user which package(s) to target
• Treat each targeted package as an independent analysis unit

Output: Project summary table — language, framework, source dirs, test dirs, file counts, monorepo status.

Phase 2: Stack Detection

Auto-detect the test stack using signals from "$SKILL_HOME"/references/framework-detection.md.

1. Scan for config files and dependency declarations
2. Determine: language, test framework, runner command
3. If multiple test frameworks detected (e.g., Jest for unit + Playwright for E2E), identify each separately
4. If --test-cmd is provided, use it as override
5. Present detection results to user and ask for confirmation before proceeding
6. Detect optional recording tools (when --evidence != off):
   • Check vhs --version — terminal test recording. If missing: warn + show install command per OS
   • Check ffmpeg -version — native screen recording + GIF conversion. If missing: warn
   • For Playwright projects: note that video can be enabled via config (video: 'retain-on-failure')
   • These are optional enhancements — baseline file evidence is always captured without tools
7. Determine app type (or use --app-type override):
   • terminal — Jest, Vitest, Mocha, Pytest, Go test, Cargo test, etc. (default for most)
   • browser — Playwright or Cypress detected as test framework
   • native — WinAppDriver, XCUITest, Appium, Robot Framework detected, or --app-type=native
   • Present tool availability and app type in the detection summary
8. Detect coverage tooling — check for existing coverage infrastructure:
   • Pre-commit hooks: .husky/pre-commit containing coverage commands, lint-staged with coverage
   • CI pipelines: .github/workflows/*.yml, .gitlab-ci.yml, Jenkinsfile — look for coverage gates/thresholds
   • Runner config: Jest coverageThreshold, pytest --cov-fail-under, Go -coverprofile in scripts
   • Third-party: Codecov config (codecov.yml), Coveralls, SonarQube
   • If none detected, flag for recommendation in the report's "Next Steps"

Output: Detection summary — runner command, detected frameworks, app type, recording tools, coverage tooling status.

Phase 3: Preflight Safety

Before running any tests, check for potential side effects.

1. Scan test configuration files for:
   • Database connection strings (check for prod-like URLs)
   • Environment variable requirements (process.env, os.environ, etc.)
   • API keys / secrets references
   • Network calls to external services
   • File system write operations outside temp dirs
2. Scan test files in scope for:
   • Direct HTTP calls to non-localhost URLs
   • Database mutation operations (INSERT, UPDATE, DELETE, DROP)
   • Payment/billing API references
   • Email/SMS sending functions
3. Classify risks:
   • Safe: Mocked/stubbed, localhost only, in-memory DB
   • Warn: External API calls, real DB mutations, file writes
   • Block: Production URLs, payment APIs, destructive operations

If any Warn/Block risks found:

• List each risk with file:line evidence
• Ask user for explicit confirmation to proceed
• Suggest mitigations (e.g., "set DATABASE_URL to test DB", "mock the payment client")

If --run=false: Skip this phase entirely. If --dry-run: Show the safety report but do not prompt for confirmation.

Phase 4: Test Execution

Run the existing test suite and capture results into the evidence package.

If --run=false, skip to Phase 6. If --dry-run, show the commands that would run and skip to Phase 6.

1. Create evidence directory structure:

   <evidence-dir>/<YYYY-MM-DD-HHmm>/
     execution/
     failures/
     gaps/
     remediation/
     recordings/
   

2. Run the test command detected in Phase 2 (or --test-cmd)

3. Always capture stdout+stderr to execution/test-output.txt (use 2>&1 | tee)

4. Always generate execution/test-summary.json by parsing the test output — see "$SKILL_HOME"/references/evidence-tools.md for the format spec

5. Apply --timeout — if execution exceeds the limit, kill the process tree and report partial results

6. Parse from output:

   • Total tests, passed, failed, skipped, duration
   • Full failure output for each failing test

7. If tests fail to start (missing deps, config error), diagnose the setup issue and report it. Do not proceed to Phase 5 failure triage — instead report the setup issue and ask user how to proceed.

8. Recording capture (when --evidence != off). Strategy depends on app type detected in Phase 2. See "$SKILL_HOME"/references/evidence-tools.md Tier 2 section for tool details.

   Terminal tests:

   • Generate a VHS tape file wrapping the test command
   • Run vhs <tape-file> instead of the raw command — outputs .gif to recordings/
   • If VHS unavailable: fall back to raw command (stdout/stderr still captured in execution/)

   Browser UI tests:

   • Playwright: temporarily set video: 'retain-on-failure' (or video: 'on' if --evidence=all)
   • Cypress: enable video: true in config
   • After test run: collect .webm files into recordings/
   • For failed tests with a known URL: npx playwright screenshot <base-url><route> <output>.png --full-page

   Native desktop app tests:

   • Before test: start ffmpeg screen recording as background process (platform-specific: gdigrab/avfoundation/x11grab)
   • Run the test command normally
   • After test: stop ffmpeg (SIGINT / taskkill on Windows), save to recordings/
   • If ffmpeg unavailable: fall back to platform-native screenshot

Output: Test execution summary table. Evidence artifacts listed if captured.

Phase 5: Failure Triage

Classify every failure from Phase 4. For each failing test:

1. Create evidence folder failures/NNN-<test-name>/ containing:

   • error-output.txt — full error message + stack trace extracted from test output
   • source-context.txt — source code around the failing line (+/- 10 lines)
   • test-code.txt — the complete failing test function/block
   • rerun-output.txt — output from rerunning the single test (flaky check)
   • screenshot.png — for browser tests, capture the page under test

2. Classify the failure:

3. For each failure, produce:
   • Category label
   • Confidence (high/medium/low)
   • Evidence link: [evidence](failures/NNN-<test-name>/)
   • Suggested fix (one sentence)

Output: Failure triage table with evidence links.

Phase 6: Gap Analysis

The core value of this skill. Identify untested or under-tested code. Uses an agent team for parallel analysis when multiple lens groups are active.

Read "$SKILL_HOME"/references/qa-team-roles.md for teammate definitions, output formats, and the synthesis protocol.

Phase 6a: Prepare (Lead)

Build the source-to-test mapping that all teammates will use:

1. Build a source-file-to-test-file mapping:
   • Convention-based: src/foo.ts -> test/foo.test.ts, src/foo.py -> tests/test_foo.py
   • Config-based: check testMatch, testPathPattern, pytest testpaths
2. Find source files with NO corresponding test file
3. Determine which teammate groups are needed based on --lens:
   • Code Quality group: functional, resilience, idempotence
   • Security & Infra group: security, infra, network
   • User & System group: frontend, journey, performance, observability
4. Map each active --lens value to its group. Only groups with at least one active lens will be spawned.
5. Assign non-overlapping gap NNN ranges per group (001-199, 200-399, 400-599)

Phase 6b: Parallel Lens Analysis (Team)

If only 1 group has active lenses (e.g., --lens=security), skip team creation and run the analysis inline — apply the QA lenses per "$SKILL_HOME"/references/qa-perspectives.md sequentially to avoid orchestration overhead.

If 2 or more groups are active, create an agent team to analyze test gaps from different quality perspectives:

1. Create the team and spawn each needed teammate with their context package (see "$SKILL_HOME"/references/qa-team-roles.md for the full spawn prompt spec):

   • Source-to-test mapping from Phase 6a
   • Project metadata (language, framework, test runner, source/test dirs)
   • Their assigned lenses and lens definitions from "$SKILL_HOME"/references/qa-perspectives.md
   • Gap NNN range, evidence directory path, filter settings (--severity, --exclude, --app-type, --base-url)

2. Each teammate independently:

   • Greps source files for their lens-specific patterns
   • Checks if matching code has corresponding test coverage
   • Generates proof files gaps/gap-NNN-<lens>-<severity>.md containing: full source snippet, explanation, pattern matched, suggested test description (see "$SKILL_HOME"/references/evidence-tools.md Tier 1 for the proof template)
   • For browser apps (--app-type=browser): the User & System teammate captures screenshots as gaps/gap-NNN-screenshot.png
   • Returns a structured findings table to the lead

3. The lead waits for all teammates to complete and report back.

Phase 6c: Synthesis (Lead)

Aggregate all teammate findings into a unified gap analysis:

1. Collect all findings tables from each teammate
2. Deduplicate — if two lenses found the same file:line, keep both findings but note the overlap
3. Apply --severity filter — drop findings below the minimum severity
4. Apply --max-findings cap — keep highest severity findings first
5. Apply --exclude pattern — skip matching files
6. Renumber gap files to a clean unified sequence (001, 002, ...) by renaming files from teammate ranges
7. Clean up the team
8. For each finding, produce a summary row:
   • Lens category
   • Severity: critical / high / medium / low
   • Confidence: high / medium / low
   • Source location: file:line
   • Pattern matched (what was detected)
   • Proof link: [proof](gaps/gap-NNN-<lens>-<severity>.md)
   • Suggested test description (what a test should verify)

Output: Gap analysis findings list, grouped by lens, with proof links.

Phase 7: Remediation & Report

7a: Write Missing Tests (only if --fix is enabled)

For each gap finding queued for remediation:

1. Determine the target test file:
   • Use the source-to-test mapping from Phase 6
   • If no test file exists, create one following the project's test naming convention
2. Write the test:
   • Follow existing test patterns in the project (imports, describe/it structure, fixtures)
   • Use deterministic assertions only — no random data, no timing-dependent checks
   • Mock external dependencies (network, DB, file system) following existing mock patterns
   • Include a comment: // QA-ENGINEER: covers <lens> gap in <source-file>:<line>
3. Run the impacted test file to verify the new test passes
4. Capture evidence to remediation/NNN-<test-file>/:
   • added-tests.diff — diff of the test file changes
   • test-output.txt — output from running the new tests
5. If the test fails:
   • If it's a test bug (assertion wrong), fix it and rerun once
   • If it's a real defect (code is actually broken), keep the test but mark it with // TODO: real defect — <description>
   • If it's flaky after 2 reruns, delete it and report the issue
6. Constraints:
   • Only edit test files — NEVER edit source/production code
   • Only add new test cases — never modify or delete existing tests
   • Maximum 10 new test files per run (ask user to continue if more needed)

7b: Generate Quality Report

Generate a markdown report following "$SKILL_HOME"/references/report-template.md.

1. Save the report as REPORT.md inside the evidence package directory (<evidence-dir>/<timestamp>/REPORT.md)
2. The evidence directory was created in Phase 4 (or create it now if --run=false)
3. Add qa-evidence/ to .gitignore if not already present (ask user first)

Report contents:

• Evidence Package manifest table (directory -> contents -> count)
• Executive summary with health verdict
• Test execution results from Phase 4 (linked to execution/ files)
• Failure triage from Phase 5 (linked to failures/NNN/ folders)
• Gap analysis findings from Phase 6, organized by lens and by severity (linked to gaps/ proof files)
• Remediation summary from Phase 7a (linked to remediation/NNN/ folders, if --fix was used)
• Coverage Tooling section — detected tools or recommendation to set up coverage gates
• Recommended next steps (include coverage tooling setup if none detected)

4. Present the executive summary to the user in chat — include:
   • Health verdict and key metrics
   • Evidence package path and file counts: "Evidence package: ./qa-evidence/<timestamp>/ — N failure folders, N gap proofs, N recordings"
   • Don't make them open the file for the headline result

7c: Recording Conversion (only when recording artifacts exist in recordings/)

1. Present recording summary: list all captured artifacts with type, size, duration
2. Ask user: "Which recording artifacts should be converted to GIF for PR?"
3. For user-approved items:
   • .webm / .mp4 -> GIF via ffmpeg palette method (see "$SKILL_HOME"/references/evidence-tools.md)
   • VHS .gif output — copy as-is (already GIF)
   • .png screenshots — keep as-is
4. Save GIFs to recordings/gif/
5. Output: "Ready for PR: N GIFs saved to recordings/gif/"

Retrospective

After completing the workflow, reflect on the entire execution session:

1. Consider: Were there mid-session corrections? Rejected outputs? Plan changes? Errors?
2. Ask the user: "Quick feedback on this run? (1-5 rating, note any issues, or press enter to skip)"
3. If the user provides feedback OR if corrections/issues occurred during this session: a. Create feedback/ directory if it does not exist (header: # Feedback Log + blank line + <!-- Append new entries at the top. Do not edit previous entries. -->) b. Prepend a new entry after the header:

   ## <ISO-8601 timestamp>
   - **Skill Version**: <version from this file's frontmatter>
   - **Task**: <brief description>
   - **Outcome**: success | partial-success | failure | error
   - **Rating**: <N>/5 (or "—" if not provided)
   - **Corrections**: <mid-session corrections, or "none">
   - **Issues**: <specific problems, or "none">
   - **User Note**: <user's verbatim feedback, or "—">
   ---
   

4. If the user skips AND no corrections or issues occurred, end without recording.

Monorepo Behavior

When a monorepo is detected:

1. Ask user which packages to target (or accept --scope=packages/foo)
2. Run Phases 2-7 independently per package
3. Generate one evidence package per package, plus a combined executive summary
4. Cross-package integration gaps are noted but not deeply analyzed (suggest using --lens=network for API boundaries)

Error Handling

Integration with Other Skills

Example Invocations

Full QA audit

Run a full QA analysis on this project

Security-focused audit, no test execution

Analyze test gaps --run=false --lens=security

Fix failing tests and fill gaps

QA this codebase --fix

Audit only changed files

Run QA on changed files --scope=changed --lens=functional,security

Dry run to preview

QA audit --dry-run

Custom test command with timeout

Run tests and analyze gaps --test-cmd="npm run test:unit" --timeout=120

QA with recording evidence for all tests

QA this project --evidence=all

Native desktop app testing

Run QA --app-type=native --evidence=on-failure

Browse evidence after a run

# Evidence package structure:
./qa-evidence/2026-03-07-1430/
  REPORT.md                          # Start here — index into all evidence
  execution/test-output.txt          # Full test output
  execution/test-summary.json        # Parsed results
  failures/001-auth-login/           # Failure evidence folder
    error-output.txt
    source-context.txt
    test-code.txt
    rerun-output.txt
  gaps/gap-001-security-critical.md  # Gap proof with source snippet
  gaps/gap-001-screenshot.png        # UI screenshot (browser apps)
  remediation/001-auth-test/         # Added test diff + output
    added-tests.diff
    test-output.txt