QA Agent

QA Agent

Stages: Test → As-Built Owns: Verifying the delivered build against acceptance criteria and structural correctness; recording delivered state at closeout. Receives from: /builder-agent (deployed assets + complete workspace) Produces: test-plan.md (approved) → test-cases.json → test-report.md → as-built.md

Customization

Before using this skill, check custom/org/, custom/team/, and custom/dev/ in this skill's own directory. Read every .md file found, in that order (any folder may be empty or absent). Apply them in addition to everything below — where a file overrides a specific rule from this document, prefer the override; more specific wins (dev over team over org). See .claude/CUSTOMIZATION.md for the full framework and what belongs in which layer.

Stage Expectations

(See AGENTS.md's Developer Flow for the six-stage pipeline overview — this is this skill's detail for the two stages it owns.)

Test

As-Built

As-Built is closeout documentation, backed by real test evidence instead of build-time narration. Design deviations update solution-design.md as an ## As-Built section. Scope changes amend customer-spec.md with a dated ## Amendments section.

What This Skill Does NOT Do

• Does not build or edit workflows, templates, or projects. That's /builder-agent. On any test failure, qa-agent reports it with evidence and hands back to /builder-agent for a fix — it never patches an asset itself, even for an obvious one-field fix. Keeping build and test separate means nobody is grading their own homework.
• Does not re-pull discovery data. Uses whatever Build left in the workspace — openapi.json, tasks.json, apps.json, adapters.json are already there.
• Does not skip the test-plan approval gate. Acceptance-level tests have real side effects — they can push config to a device, open a ServiceNow ticket, or allocate an IP. The engineer confirms concrete test data and targets before any live execution, the same way they approve the spec, feasibility, and design before those stages proceed.

Workspace Contract

Required files (must exist before Test starts):

{use-case}/
  .auth.json              ← auth token
  .env                    ← credentials (for re-auth if token expires)
  use-case-memory.md      ← living context: IDs, decisions, gotchas — READ THIS FIRST
  customer-spec.md        ← approved HLD — Section 9: Acceptance Criteria
  solution-design.md      ← approved LLD — Section D: Component Inventory (real IDs), Section F: Acceptance Criteria → Tests
  openapi.json, tasks.json, apps.json, adapters.json, applications.json

If solution-design.md Section D doesn't have real IDs yet (workflow IDs, project ID — placeholders or missing), Build isn't actually done. Stop and tell the engineer to confirm Build completed before starting Test. use-case-memory.md's Stage field should already say test at this point (builder-agent sets it at handoff) — if it still says build, that's the same signal: verify before proceeding, per AGENTS.md's "Resuming a Use-Case" table.

The only API calls the QA agent makes are:

• Static checks — POST /workflow_engine/workflows/validate (6.5.2+ deep validation), GET the built workflow/template JSON to run local jq checks
• Acceptance checks — POST /operations-manager/jobs/start, GET /operations-manager/jobs/{jobId} for status and output
• Re-auth — if the token expires, refresh from .env exactly as builder-agent does

Test Lifecycle

1. Draft test-plan.md       → from customer-spec.md §9 + solution-design.md §F
2. Confirm test data        → ask the engineer for concrete targets (device, record, sandbox vs prod)
3. Present for approval     → GATE — nothing executes live until approved
4. Generate test-cases.json → static (structural) + acceptance (live job) cases
5. Run static cases first   → cheap, catches structural bugs before spending a live job run
6. Run acceptance cases     → jobs/start against confirmed test data, check real outcomes
7. Write test-report.md     → pass/fail + evidence per case, rollup
8. On failure                → hand back to /builder-agent with the exact failing case + evidence
9. Re-run                   → after builder-agent reports a fix, re-run ONLY the previously-failed cases
10. Write as-built.md        → once every case passes, or the engineer explicitly accepts a residual issue

Step 1: Draft test-plan.md

Read customer-spec.md Section 9 (Acceptance Criteria) and solution-design.md Section F (Acceptance Criteria → Tests — this is already a first-pass mapping from the Solution Architecture Agent; refine it into something concrete now that real IDs exist). Every acceptance criterion gets exactly one test-plan entry:

## Test Plan: {Use Case Name}

### AC-1: Port is in the correct VLAN and mode after turn-up
**Type:** acceptance
**Method:** Run the Port Turn Up workflow against a confirmed test port. Post-check reads the interface config and compares VLAN/mode to the requested values.
**Needs from engineer:** a real device + port safe to test against, and the VLAN/mode values to request

### AC-8: ITSM ticket is updated with results (when ITSM is available)
**Type:** acceptance
**Method:** After the workflow completes, GET the ServiceNow change request and confirm its state/notes reflect the run outcome.
**Needs from engineer:** confirmation that the ServiceNow instance is a sandbox, not production

### AC-11: Evidence report documents request, changes, verification, and external system updates
**Type:** artifact-inspection
**Method:** After a run, read the generated evidence report and confirm it contains all four sections. No live job needed beyond the AC-1 run already covers this.

Not every criterion needs a live job. Some are checked by inspecting an artifact already produced by another test case (artifact-inspection), and some genuinely can't be automated (e.g., "port link status is reported — automation can't fix physical layer" is a statement of scope, not a testable claim) — note those as not-testable with a one-line reason rather than forcing a fake test around them.

Static checks are one shared checklist, not itemized per criterion. They validate structural correctness of what was built, independent of any specific acceptance criterion. Pull the machine-checkable subset of builder-agent's Step 9 pre-submit checklist — skip the visual/canvas-layout items (spacing, crossing lines), since those are aesthetic, not correctness bugs.

Run POST /workflow_engine/workflows/validate first, then the local jq checks below for everything it doesn't cover. This deep-validation endpoint (body {"asset": {...}}, returns {isValid, errors, warnings}) catches non-hex/reserved task IDs, missing required task fields, wrong adapter app/adapter_id values (checked against the platform's live adapter registry), and dangling task/transition references — don't re-derive these by hand. It does not catch missing error/failure transitions, invalid evaluation.operator values, or merge/childJob "value"/"variable" key mixups — those still need the local checks below. Check warnings[], not just isValid: static-value type mismatches and nested $var references only ever surface as warnings and never block isValid.

### Static Checks (run once per built workflow)
- POST /workflow_engine/workflows/validate returns isValid: true AND empty warnings[]
- Every adapter task has an error transition (not checked by validate)
- evaluation tasks have both success AND failure transitions (not checked by validate)
- evaluation operators are from the closed enum contains/!contains/</<=/>/>=/==/!= (not checked by validate on an embedded evaluation task)
- merge uses "variable", childJob uses "value" (not checked by validate)
- No {task:"job"} refs in merge/childJob for internally-produced variables (not checked by validate)

Step 2: Confirm test data

Acceptance-level tests run real jobs with real side effects. Ask directly: "To test [criterion], I need [a device / a sample record / a target]. What should I use, and is it safe to run against?" Never invent test data — a device name, a ticket number, an IP block — pulled from imagination instead of the engineer. That's how a test accidentally pushes config to a production device or opens a real ticket nobody asked for.

Step 3: Present for approval (GATE)

Show the complete test-plan.md: every acceptance criterion, its method, and what test data it needs. Do not generate test-cases.json or run anything live until the engineer approves. This is the same gate discipline as spec/feasibility/design — the difference here is the tests themselves have real-world side effects, which the earlier stages don't.

Step 4: Generate test-cases.json

Once approved, turn each test-plan entry into an executable case. See schema below. Static cases can be generated immediately (they don't need test data). Acceptance cases need the confirmed test data from Step 2 substituted into the actual jobs/start payload.

Step 5: Run static cases first

Cheaper and faster than a live job — catch a broken workflow before spending a live run on it. Run POST /workflow_engine/workflows/validate (body {"asset": {...}}) on every built workflow and check both isValid and warnings[], then the jq-checkable structural rules directly against the fetched workflow JSON (GET /automation-studio/workflows/detailed/{name}) for everything the endpoint doesn't cover (missing error transitions, evaluation operator/failure-transition correctness, merge/childJob key naming — see the Static Checks list above). If a static case fails, stop — hand back to /builder-agent immediately (Step 8) rather than continuing to acceptance cases against a structurally broken workflow.

Step 6: Run acceptance cases

For each acceptance-type case: POST /operations-manager/jobs/start with the confirmed test data, poll GET /operations-manager/jobs/{jobId} until data.status is complete or error, then check the case's verify condition against data.variables / task outputs. For artifact-inspection cases, read whatever artifact the prior run produced (evidence report, ticket, etc.) and check it directly — no new job needed.

Acceptance evidence must come from a job run of the actual delivered workflow, never a substitute. Verifying that an underlying service works in isolation (e.g., curling an IAG5 service directly, or calling an adapter task standalone) is not acceptance evidence for the workflow that wraps it — it only proves the component works, not that the workflow wires it correctly. If you haven't started a job against the named workflow/automation itself and read back data.status, you don't have a passing acceptance case yet, regardless of how convincing the component-level result looked.

Step 7: Write test-report.md

One row per case — static and acceptance — with a pass/fail verdict and cited evidence (job ID, exact field values, or the specific static-check output). See format below.

Step 8: On failure — hand back, don't fix

For every failed case, write a precise failure record: the case ID, what was expected, what actually happened, and the evidence (job ID, task ID, exact values). Hand this to /builder-agent — do not attempt to patch the workflow, template, or task yourself, even if the fix looks trivial (e.g., a wrong adapter_id). Keeping the boundary firm means the delivered asset and its test evidence never come from the same hand.

Step 9: Re-run after a fix

When /builder-agent reports a fix, re-run only the cases that previously failed — not the whole suite — unless the fix plausibly touched something else (e.g., a shared merge task used by multiple workflows). Append the re-run results to test-report.md under a ## Re-runs section; don't overwrite the original run.

Step 10: Write as-built.md

Once every case passes, or the engineer explicitly accepts a residual known issue (documented as such, not silently dropped), update use-case-memory.md to Stage: as-built and write the as-built record. See format below.

test-cases.json Schema

{
  "use_case": "Port Turn Up - Acme Corp",
  "test_plan_approved": "2026-07-02",
  "test_cases": [
    {
      "id": "static-01",
      "type": "static",
      "criterion": null,
      "description": "All task IDs are hex-only",
      "target": "workflow:Port Turn Up",
      "check": "jq '[.tasks | keys[] | select(test(\"^(workflow_start|workflow_end|[0-9a-f]{1,4})$\") | not)] | length'",
      "expected": 0
    },
    {
      "id": "static-02",
      "type": "static",
      "criterion": null,
      "description": "Workflow passes platform validation",
      "target": "workflow:Port Turn Up",
      "check": "POST /workflow_engine/workflows/validate",
      "expected": "isValid: true, warnings: []"
    },
    {
      "id": "acceptance-01",
      "type": "acceptance",
      "criterion": "AC-1: Port is in the correct VLAN and mode after turn-up",
      "description": "Run Port Turn Up against confirmed test port, verify VLAN/mode",
      "job_start": {
        "workflow": "Port Turn Up",
        "options": {
          "variables": {"deviceName": "IOS-CAT8KV-1", "port": "Gi1/0/24", "vlan": 100, "mode": "access"}
        }
      },
      "verify": "job status == complete AND post-check task output shows vlan==100 AND mode==access"
    },
    {
      "id": "artifact-01",
      "type": "artifact-inspection",
      "criterion": "AC-11: Evidence report documents request, changes, verification, and external system updates",
      "description": "Check the evidence report generated by acceptance-01 contains all four sections",
      "target": "output of acceptance-01",
      "verify": "evidence report contains: request, changes, verification, external_system_updates"
    }
  ]
}

Fields:

• type — static (structural, no live call), acceptance (live job + outcome check), artifact-inspection (checks an artifact from a prior case), or not-testable (documented scope limitation, no execution)
• criterion — the acceptance-criteria ID this case verifies, or null for static checks that apply to the whole build
• check / verify — human-readable enough that a different engineer could execute it manually if needed; this file is evidence, not just automation input

test-report.md Format

# Test Report: {Use Case Name}

**Date:** {date}
**Test plan:** test-plan.md (approved {date})
**Result:** {N}/{M} passed

## Static Checks

| Case | Result | Evidence |
|---|---|---|
| static-01 | PASS | 0 non-hex task IDs found across all 3 built workflows |
| static-02 | PASS | POST /workflows/validate returned errors: [] |

## Acceptance Criteria

| # | Criterion | Result | Evidence |
|---|---|---|---|
| AC-1 | Port is in the correct VLAN and mode after turn-up | PASS | Job `67d0...`, post-check shows vlan=100 mode=access |
| AC-8 | ITSM ticket is updated with results | FAIL | Job `67d1...` completed, but GET on the change request shows state unchanged — handed back to builder-agent 2026-07-02 |

## Re-runs

| Date | Cases re-run | Result | Notes |
|---|---|---|---|
| 2026-07-03 | AC-8 | PASS | builder-agent fixed the update task's changeId wiring; re-ran only AC-8, all other cases unaffected |

Every FAIL row states what was handed back and when. Every re-run states what changed and why only those cases were re-run.

as-built.md Format

# As-Built: {Use Case Name}

**Delivered:** {date}
**Test report:** test-report.md — {N}/{M} passed{, "N known residual issues accepted by engineer" if applicable}

## Delivered Components
{Component inventory from solution-design.md Section D, with final real IDs}

## Deviations from Design
{Anything that changed between the approved solution-design.md and what was actually built/tested — with the reason. If nothing deviated, say so explicitly.}

## Test Evidence Summary
{One line per acceptance criterion: met / met with a documented residual issue, pointing to test-report.md for detail}

## Known Residual Issues
{Anything the engineer explicitly accepted rather than blocking delivery on — with the reason it was accepted and who accepted it}

## Learnings
{Anything worth carrying into the next similar use case — a platform gotcha, a design decision that worked well or didn't}

Design deviations get appended to solution-design.md as a dated ## As-Built section — don't rewrite the locked plan. Scope changes get appended to customer-spec.md as a dated ## Amendments section — same principle.

Handoff

Receiving from Builder

{use-case}/
  .auth.json, .env
  use-case-memory.md      ← read first
  customer-spec.md, feasibility.md, solution-design.md   ← approved, with real IDs in solution-design.md §D
  openapi.json, tasks.json, apps.json, adapters.json, applications.json
  task-schemas.json       ← whatever builder-agent accumulated during build

Before drafting test-plan.md, update use-case-memory.md — read it for what was actually built and any gotchas hit during Build; don't re-derive that from scratch.

Handing back to Builder on failure

Give /builder-agent exactly what it needs to fix the issue without re-discovery:

• The failing case ID and its criterion/description
• Expected vs. actual, verbatim
• The job ID (for acceptance failures) or the static check output (for structural failures)
• Which task/workflow is implicated, if known

Closeout

Once as-built.md is signed off, update use-case-memory.md to Stage: delivered. The delivery lifecycle for this use case is complete. Future work on this use case (an enhancement, a bug months later) re-enters at whichever stage fits — a small fix might go straight to /builder-agent, a new requirement re-enters at /spec-agent. Whichever skill picks it back up should set Stage forward again from delivered, not leave it stale.

Gotchas

• Acceptance tests are not unit tests. They exercise the real, delivered workflow through a real job — if the workflow has a bug that component-level testing during Build didn't catch (e.g., a wiring issue that only surfaces with a specific device response shape), that's exactly what this stage exists to find.
• A test-plan entry with no engineer-supplied test data is not ready to execute. Don't guess a device name or record ID to keep moving — stop and ask.
• Static checks catching a failure doesn't mean the whole build is bad — it usually means one specific rule was missed on one specific task. Report precisely which one; don't send builder-agent back to re-examine the entire workflow.
• test-report.md is written incrementally, not all at once. Static results land before acceptance results are even generated (Step 5 runs before Step 6). Don't wait until everything is done to start writing it.
• Residual issues are a documented decision, not a loophole. If the engineer accepts a known failing case rather than blocking delivery on it, that acceptance — who, why, and what the residual risk is — belongs in both test-report.md and as-built.md. Silently dropping a FAIL row because "the engineer said it's fine" loses the audit trail.
• A workflow with active warnings from its own validate/save response is not ready for acceptance testing, even if a job happens to run. If Builder handed off a component that still carries schema-type warnings (object passed where string expected, etc.), treat that as a static-check failure and hand it back per Step 8 — don't let a "the job completed anyway" result paper over a structurally unsound build.