QA Orchestration

QA Orchestration

Extends agentic-delegation with the Playwright test lifecycle. The parent skill establishes the economics (agents are cheap, orchestrator context is expensive), the model ladder (haiku-first with upgrade-on-failure), execution patterns (parallel fan-out, sequential pipeline), and quality governance (re-launch principle, contradiction resolution). This skill adds the Plan->Generate->Execute->Heal->Report loop that governs how those patterns apply to automated testing.

If the orchestration plugin is installed, read agentic-delegation's SKILL.md before proceeding. If not installed, recommend installing it. Core principles: delegate aggressively (agents are cheap), the orchestrator's context is for decisions and dispatch (never for grunt work), agents communicate through files on disk (the orchestrator routes, never relays).

Verb Interpretation

Every user verb implies agent-delegated execution. The orchestrator decomposes, dispatches, and assembles. It never executes directly.

• "run tests" / "run QA" -- dispatch the full Plan->Generate->Execute->Heal->Report loop.
• "generate tests" / "create tests" -- dispatch generator-agent (Phase 2).
• "heal tests" / "fix selectors" / "fix locators" -- dispatch healer-agent (Phase 4).
• "plan tests" / "explore the app" -- dispatch planner-agent (Phase 1).
• "execute tests" / "run the suite" -- dispatch executor-agent (Phase 3).
• "fix timing" / "fix test architecture" -- dispatch generator-agent in fix mode.

No exceptions. The orchestrator coordinates; agents act.

System Architecture

Parent Framework: agentic-delegation (orchestration plugin)

Provides: model ladder, decomposition patterns, execution patterns, quality governance (re-launch principle, contradiction resolution, concurrent file write prevention, independent verification), session persistence (ARRIVE/WORK/LEAVE lifecycle). This skill uses the parent's parallel fan-out (healing dispatch), sequential pipeline (phase ordering), status-driven branching, and file-based communication.

For multi-session QA work, the parent's session persistence protocol applies -- conventions.md holds QA-specific rules (locator strategy decisions, confidence threshold tuning, browser-specific patterns), codebase_state.md holds the test inventory and healing history, deferred_items.md tracks unresolved test failures deferred from healing.

Agents

Artifact Contract

Canonical inventory. The table is authoritative; body prose may reference rows by ID and MUST NOT contradict them. ${stem} = test filename stem; ${test-name} = sanitized test title.

Notes on producer/consumer accuracy:

• A2 (pages.md): planner-internal DOM quality output, NOT a generator input. Generator's Pre-Flight check and Reads list omit it; selector-strategy.md (A3) carries the actionable derivative.
• A15 (results.json): healer consumes the normalized form (.ai-failures.json, A16), not results.json directly.
• A22 (lessons.md): only generator-in-fix-mode and healer write it. Planner and executor do not.

Status Protocol

Each agent writes .playwright/orchestrator-status.json on completion:

• DONE -- phase completed successfully
• NEEDS_CONTEXT -- agent needs user input
• BLOCKED -- unrecoverable error

Phase sequence: PLAN -> GENERATE -> EXECUTE -> HEAL. Written to .claude/qa-phase.txt after each phase completes. Read on session resumption to determine re-entry point.

Multi-App Mode

Active when multiple base URLs are detected. Phase adaptations:

• Plan: Dispatch planner with all URLs. Produces per-app artifacts: .playwright/pages-{app-name}.md, .playwright/selector-strategy-{app-name}.md.
• Generate: Dual-context fixtures (one browser context per app). Comparison-as-findings pattern (log deltas, not absolute assertions).
• Heal: Per-app locator healing. app_specific field identifies which app's DOM changed.
• Report: Finding summary table replaces pass/fail for comparative tests.

Finding taxonomy:

Findings are data for humans to interpret, not bugs for the healer to fix.

Constraints

• Retry a failed phase once, then stop.
• Max 3 PRs per session.
• Preserve .claude/qa-phase.txt across failures for session resumption.
• The orchestrator never edits test files directly. Agents own all file mutations.

Conflict Resolution

When a QA spec contradicts a project principle, the project principle wins. Flag the spec for correction. Document the conflict in .playwright/lessons.md with: the spec recommendation, the principle it violated, and the resolution applied. Specs serve projects, not the reverse.

Generator-Agent Dispatch Modes

The generator-agent operates in three modes:

Modify mode exists because the generator's create/fix protocol adds overhead (plan reading, verify loop) that provides no value for targeted edits like "add 20 lines to this test block." The modification prompt contains all context needed.

Quality Gates

Three gates at three scopes:

The QA Loop

Prerequisites -> Plan -> (Generate -> Execute -> Heal)* -> Report

The orchestrator drives this loop. The outer sequence is linear (Plan then inner loop then Report). The inner loop (Generate->Execute->Heal) repeats until an exit condition is met. Each phase dispatches one agent, reads its output files, and branches on status.

Inner Loop Exit Conditions

Exit the Generate->Execute->Heal cycle when any of these is true:

• All tests pass. Zero failures of any category.
• Only structural failures remain. All locator failures healed; remaining failures are timing, assertion, or infrastructure issues requiring human intervention.
• Circuit breaker trips. Max healing attempts per test exceeded, or max PRs per session reached.
• No progress. A healing round produced zero successful fixes (same failures persist). Stop to prevent infinite looping.

Prerequisites Check

Before any phase:

1. Check tests/seed.spec.ts exists. If missing, STOP. Tell the user a seed file is required. Seed must contain 5 components: import statement, navigation, interaction, assertion, accessibility-first locators.

2. Check the base URL. Extract from the user's argument, or check .playwright/project-config.md. If neither exists, ask the user.

3. Session resumption. Read .claude/qa-phase.txt if it exists. Resume from the phase after the one recorded (PLAN -> resume at GENERATE, etc.). If missing, start from Phase 1.

Phase 1: PLAN

Record the dispatch start time: T0=$(date +%s). Dispatch planner-agent with the base URL.

After completion, read .playwright/orchestrator-status.json and branch:

• DONE -> run Planner Completeness Check AND CLI Evidence Check, then write PLAN to .claude/qa-phase.txt, proceed to Phase 2
• NEEDS_CONTEXT -> surface the blocker field to the user, re-dispatch once after input
• BLOCKED -> surface blocker, stop

Planner Completeness Check

Verify all 5 required artifacts exist:

1. .playwright/test-plan.md
2. .playwright/pages.md
3. .playwright/selector-strategy.md
4. .playwright/project-config.md
5. .playwright/VERIFICATION.md

If any are missing, re-dispatch planner-agent with explicit instructions to produce the missing artifacts.

CLI Evidence Check

The planner explores via @playwright/cli, which writes accessibility snapshots to .playwright/snap-*.yaml. Snapshot absence proves non-CLI exploration (MCP-only, scripted, or fabricated from memory). Verify:

find .playwright -maxdepth 1 -name 'snap-*.yaml' -newermt "@$T0" | head -1

• One or more matches -> CLI evidence present, proceed.
• Zero matches (first miss) -> re-dispatch planner-agent with this added directive: "MANDATORY: explore via @playwright/cli only. Each visited page MUST produce a .playwright/snap-<page>.yaml snapshot. Scripts that launch browsers are prohibited." Reset T0 before re-dispatch.
• Zero matches (second miss) -> escalate: write NEEDS_CONTEXT to .playwright/orchestrator-status.json with blocker "planner produced no CLI snapshots across two dispatches; manual investigation required" and stop.

NEVER mark Phase 1 complete on agent self-report alone. NEVER skip the CLI Evidence Check. NEVER accept a missing snapshot set with the rationale "the agent had a reason" — silent skip is the failure mode this check exists to catch.

Phase 2: GENERATE

Snapshot the pre-dispatch test set: BEFORE=$(ls tests/*.spec.ts 2>/dev/null | sort). Record T0=$(date +%s). Dispatch generator-agent. Include .playwright/lessons.md path in the dispatch if it exists (agents read it before starting -- it contains discoveries from prior cycles).

After completion, read .playwright/orchestrator-status.json and branch:

• DONE -> run Generator Live-Verification Check, then write GENERATE to .claude/qa-phase.txt, proceed to Phase 3
• NEEDS_CONTEXT -> surface blocker, re-dispatch once
• BLOCKED -> surface blocker, stop

Generator Live-Verification Check

The generator writes one snapshot per test to .playwright/snap-gen-${stem}.yaml (artifact A14). The snap-gen- prefix disambiguates from planner snapshots (A6). For each new test in tests/${stem}.spec.ts, a matching snap-gen-${stem}*.yaml MUST exist with mtime after T0:

AFTER=$(ls tests/*.spec.ts 2>/dev/null | sort)
NEW=$(comm -13 <(echo "$BEFORE") <(echo "$AFTER"))
for t in $NEW; do
  stem=$(basename "$t" .spec.ts)
  find .playwright -maxdepth 1 -name "snap-gen-${stem}*.yaml" -newermt "@$T0" | head -1
done

• Every new test has a matching snapshot -> verification evidence present, proceed.
• Any new test lacks a snapshot (first miss) -> re-dispatch generator-agent with this added directive: "MANDATORY: for each test in tests/<name>.spec.ts, run the locator-verification step and persist the snapshot to .playwright/snap-gen-<name>.yaml. Tests written without a corresponding snapshot will be rejected." Reset T0 and BEFORE before re-dispatch.
• Any new test still lacks a snapshot (second miss) -> escalate: write NEEDS_CONTEXT to .playwright/orchestrator-status.json with blocker "generator skipped live verification across two dispatches for tests: <list>; manual review required" and stop.

Generator Conditional-Artifact Check

The generator's status file (A7) lists produced artifacts in its artifacts field. For every conditional path it claims (A11 tests/pages/*.page.ts, A12 tests/fixtures.ts, A13 tests/helpers/test-data.ts), confirm the file exists on disk:

for path in $(jq -r '.artifacts[]?' .playwright/orchestrator-status.json); do
  case "$path" in tests/pages/*|tests/fixtures.ts|tests/helpers/*) test -s "$path" || echo "MISSING: $path";; esac
done

• All claimed artifacts present -> proceed.
• Any missing (first miss) -> re-dispatch generator-agent with the missing list as a NEEDS_CONTEXT correction directive.
• Any missing (second miss) -> escalate: write NEEDS_CONTEXT with blocker "generator claimed artifacts not on disk across two dispatches: <list>" and stop.

Orchestrator MUST run both checks before advancing to Phase 3. Orchestrator MUST NOT mark Phase 2 complete on agent self-report alone. NEVER accept a missing snapshot or a missing claimed artifact — the snapshot is the audit trail; the claimed-artifact list is the contract.

Phase 3: EXECUTE

Record T0=$(date +%s). Dispatch executor-agent. Include .playwright/lessons.md path if it exists.

After completion, read .playwright/orchestrator-status.json and run the Executor Output Check before reading .ai-failures.json.

Executor Output Check

The executor's load-bearing artifact is .ai-failures.json (A16) — Phase 4 routing reads it. Verify:

test -s .ai-failures.json && jq -e '.summary and (.locator | type == "array")' .ai-failures.json >/dev/null
find . -maxdepth 1 -name results.json -newermt "@$T0" | head -1

• Both checks pass -> proceed to Failure Routing.
• .ai-failures.json missing or schema-invalid (first miss) -> re-dispatch executor-agent with directive: "MANDATORY: write .ai-failures.json with the full schema (summary + per-category arrays). Status DONE without this file is invalid." Reset T0 before re-dispatch.
• Still missing or invalid (second miss) -> escalate: write NEEDS_CONTEXT to .playwright/orchestrator-status.json with blocker "executor produced no valid .ai-failures.json across two dispatches" and stop.

Orchestrator MUST NOT advance to Phase 4 without a valid .ai-failures.json. NEVER infer failures from orchestrator-status.json alone — its healable_count field is a summary, not the routing source of truth.

Failure Taxonomy

Six categories by regex on error messages. Only locator failures are healable.

Failure Routing Decision Tree

Test fails ->
  Assertion/data match? -> DO NOT HEAL (real bug)
  Locator match? -> Verify: element found but assertion failed? Reclassify as DATA.
                    Console ERROR/FATAL? Reclassify as REAL BUG.
                    10+ concurrent failures? Reclassify as INFRASTRUCTURE.
                    Confirmed locator -> proceed to Phase 4
  Timing match? -> Re-dispatch generator-agent in fix mode (fix waits, not locators)
  Other? -> Manual investigation

Test Completeness Gate

Before routing failures, check that passing tests are substantive. A test that navigates and logs "needs implementation" without asserting is a hollow pass. If passing tests lack real assertions (no expect(), no comparison logic), re-dispatch generator-agent to complete the hollow tests before proceeding.

Failure Routing

• DONE with zero failures -> check exit conditions, proceed to Report or re-enter inner loop
• DONE with locator failures -> proceed to Phase 4 (Heal)
• DONE with timing failures where locators are correct but test structure is wrong (serial execution of independent tests, missing waits, fixture teardown races) -> re-dispatch generator-agent in fix mode with failing test paths and error descriptions
• NEEDS_CONTEXT -> surface blocker, re-dispatch once
• BLOCKED -> surface blocker, stop

Write EXECUTE to .claude/qa-phase.txt.

Phase 4: HEAL

Read .ai-failures.json. Count locator-category entries (N). Record T0=$(date +%s).

Pre-Dispatch Circuit-Breaker Gate

Healers read .github/healing-state.json (A21) for blocklist + attempt counts. The orchestrator tracks the inner-loop round in its own context (round 1 = first entry into Phase 4 this session; round 2+ = re-entries after a prior heal cycle).

• Round 1: file may be absent — healers treat absence as empty. Proceed to Dispatch.

• Round 2+: the prior round's Aggregate and Route step MUST have persisted this file. Verify before dispatch:

  test -s .github/healing-state.json && jq -e '.healing_attempts' .github/healing-state.json >/dev/null
  

  If the file is absent or malformed in round 2+, abort the inner loop and surface to the user: the aggregation step failed to persist circuit-breaker state. Orchestrator MUST NOT dispatch healers in round 2+ without an authoritative state file.

Dispatch

• N == 0: Skip healing. Check exit conditions.
• N < 5: Single healer-agent (batch mode) with the full .ai-failures.json path and .playwright/lessons.md. Expected output: .healing-results.json (A19).
• N >= 5: Parallel fan-out -- one healer-agent per failure (parallel mode). Write single-item input files to .playwright/healed/{test-name}-input.json, launch all N agents in a single message. Expected output: .playwright/healed/${test-name}.json per agent (A20).

Confidence Scoring

Seven-signal weighted algorithm:

Primary signals (70% weight):

1. Error type analysis (max +0.30): ELEMENT_NOT_FOUND +0.30, ELEMENT_NOT_VISIBLE +0.25, ASSERTION -> 0.0
2. DOM change detection (max +0.20): attribute changes +0.20, relocated +0.15, removed +0.05, unchanged -0.15
3. Console/runtime errors (max +0.20): none +0.20, WARNING +0.10, ERROR/FATAL -> 0.0

Secondary signals (30% weight): 4. Network/API health (max +0.10): OK +0.10, errors -0.10 5. Element match quality (max +0.10): similarity * 0.10, no candidate -> 0.0 6. Historical pattern (max +0.05): stable +0.05, flaky -0.10 7. Cluster analysis (max +0.05): isolated +0.05, small cluster +0.03, large (10+) -0.15

Tier adjustment: tiers 1-3 +0.05, tiers 7-10 -0.05

Hard Rejection (score = 0.0)

1. Assertion on element value -- real bug
2. Runtime errors / crashes -- application broken
3. Console errors ERROR/FATAL -- application broken
4. No candidate element found -- cannot heal
5. 10+ failures in the same test run -- infrastructure issue

Decision Thresholds

Circuit Breaker

Schema (.github/healing-state.json):

{"healing_attempts":{"file:line:title":1},"prs_created":0,"blocklist":[],"last_updated":"ISO-8601"}

Rules:

• Max 2 healing attempts per test (key: "file:line:title")
• Max 3 PRs per session
• Tests failing healing twice added to blocklist
• Blocklisted tests skipped until manually removed

Cost Model

Run deterministic healing first. LLM fallback only when deterministic fails. Max confidence for LLM fixes: MEDIUM. Never auto-apply LLM fixes.

Healer Output Check

Confidence routing depends on the healer's per-failure records (confidence, tier, original/healed locator). The status file (A7) carries summary counts only — it is NOT the routing source of truth.

case "$MODE" in
  batch)    test -s .healing-results.json && jq -e '.results | type == "array"' .healing-results.json >/dev/null ;;
  parallel) ls .playwright/healed/*.json 2>/dev/null | grep -v -- '-input\.json$' | head -1 ;;
esac

• Expected file(s) present and well-formed -> proceed to Aggregate and Route.
• Missing or malformed (first miss) -> re-dispatch the failed healer(s) with directive: "MANDATORY: write per-failure output to <expected path>. Status DONE without this file is invalid."
• Missing or malformed (second miss) -> escalate: write NEEDS_CONTEXT with blocker "healer produced no valid output across two dispatches in <mode> mode" and stop.

Orchestrator MUST NOT enter confidence routing without verified healer output. NEVER infer fix decisions from orchestrator-status.json counts alone.

Aggregate and Route

After the Healer Output Check passes:

1. Read healer output: .healing-results.json (batch) or .playwright/healed/${test-name}.json files (parallel).

2. Update circuit breaker (.github/healing-state.json, A21): increment attempts per test, blocklist tests failing twice, increment PR counter. Confirm the write succeeded before exiting the round:

   test -s .github/healing-state.json && jq -e '.healing_attempts' .github/healing-state.json >/dev/null \
     || { echo "ABORT: failed to persist healing-state.json"; exit 1; }
   

   Orchestrator MUST persist this file before re-entering the inner loop. The next round's Pre-Dispatch Circuit-Breaker Gate enforces the read side.

3. Route by confidence: HIGH -> consolidated auto-merge PR. MEDIUM -> review PR. LOW -> no PR, record as deferred.

4. PR budget: Max 3 PRs per session.

Update Lessons

After aggregating healer results, append discoveries to .playwright/lessons.md:

• Selectors that failed and why (prevents repeating same guesses)
• Patterns that worked (successful tier/locator combinations)
• Structural issues found (timing, fixture, architecture problems)

Write HEAL to .claude/qa-phase.txt. Check exit conditions -- proceed to Report or re-enter inner loop at GENERATE.

Phase 5: Report

Validate findings before reporting. Generator must not log implementation gaps (harness errors, placeholder logic) as business findings. If findings exist, verify each has non-placeholder values (not 'TODO', 'needs implementation', empty strings). Strip or flag invalid findings.

Write .playwright/session-report.md:

# QA Automation Session Report
Date | Duration | Phases Completed
## Test Results
Total | Passed | Failed | Healed | Deferred
## Healing Summary
Per-tier breakdown with confidence averages
## Deferred
Test file:line -- reason -- confidence
## Artifacts
Links to test-plan, healing-results, HTML report, healed outputs

Report summary to user. Clear .claude/qa-phase.txt.

Exclude from user-facing summary: raw JSON artifacts, trace file paths, internal status file contents.

Error Handling

If any agent dispatch fails (no output file, agent error):

• Report the failure phase and error to the user
• Preserve .claude/qa-phase.txt for session resumption
• Retry a failed phase once, then stop

If exit condition is ambiguous (hollow passes mixed with real passes): re-enter at GENERATE to complete hollow tests.

If status contradicts (phase.txt says EXECUTE but orchestrator-status.json says PLAN): surface the contradiction to the user and ask for direction.

If .playwright/orchestrator-status.json is missing after agent dispatch: treat as agent failure, report to user.

If .github/healing-state.json is corrupt or unparseable: reset to empty state {"healing_attempts":{},"prs_created":0,"blocklist":[]} and continue.