qa-automation
Author end-to-end / UI-automation tests derived from the acceptance criteria, not from
the implementation. Cover risk-tier critical journeys plus their key negative paths —
not every path.
Test against observable behavior, not the implementation's internals, and never weaken
assertions to make a test pass.
Inputs
Your instructions name what to read — the acceptance criteria, the functional test-case
catalog, and (for cross-service journeys) the contract — and the artifact path to write.
Standalone? read them and write to a path you choose (and tell the user where).
Steps
- Use the project's E2E framework, never a new one. When your instructions name the
project's E2E suite (or, standalone, the project already has one: e.g. Playwright, Cypress,
WebdriverIO, Maestro for mobile), automate in that framework, reusing its config, fixtures and
helpers, without installing packages or browsers. When the project has no E2E framework, do
not add one and write no suite: check each case directly against the running system with the
tools already there (HTTP requests, the CLI, database queries, the repositories' own test
runners), record the observed result as the case's evidence, and skip the
tasks.json and
scenario steps below. The standards below still apply to what you check and how you report.
- Source the journeys. If
test_cases_path exists, that catalog is the source of truth —
automate its cases, preserving each case's ID/traceability, and do not silently drop a case
(note any you defer and why). Only when no catalog is present do you derive journeys straight
from the acceptance criteria. Either way: rank by risk and scope to the default tier —
critical-journeys-only; pick the critical
ones and the highest-value negative paths; state what is out of scope.
- Design test data — each test seeds and tears down its own data; no shared mutable state.
- Author tests with stable selectors (roles/test-ids), explicit waits on conditions,
and assertions that verify observable behavior + the contract's outcomes.
- Emit the scenario DAG — write a
tasks.json (see section below) so the
suite can be authored in parallel.
- Run in a clean env — when your instructions say the environment is already running
(the caller started it from the code under test), seed → run against it, and leave starting
and stopping to the caller. Otherwise
stack up → seed → ready_check → run → stack down
(teardown always runs).
- On failure, apply the fix-loop rule (selectors/flakes/waits only) — one attempt per
invocation; the workflow's
max_visits on the fix node (typically 3) bounds the loop.
Standards every suite must satisfy
- Risk-tiered — critical journeys + key negatives; explicit out-of-scope note.
- Isolated & idempotent — self-contained setup/teardown; order-independent.
- Deterministic — wait on conditions/selectors, never fixed sleeps; stable selectors,
not brittle CSS chains; control time/randomness where possible.
- Cross-service — real contract, not mocks, in the clean stack for cross-service journeys.
- No weakened assertions — self-healing may refresh selectors / re-run only; it must NOT
weaken assertions, permissions, or expected behavior. Approved tests are frozen.
- Safe — no real secrets or production data; seeded test data only.
- Reportable — on failure, capture screenshot/trace/video and the failing step.
Edge cases / journeys to include
- Primary happy journey end-to-end across services.
- Validation failures surfaced to the user; permission-denied journey.
- Empty state and "many items" / pagination boundary.
- Slow/failed API handling (retry/error UI); session expiry mid-flow.
- Concurrency/double-submit where the journey allows it.
- Mobile: gestures, back button, deep links, orientation (Maestro).
External skill (provision — test generation)
If a suitable test-generator skill is installed you may delegate test generation to it —
the output must still meet the standards above (isolation, determinism, real assertions,
risk-tiering). Otherwise author in-pack.
Emit tasks.json (parallel scenario authoring)
Write a tasks.json under your QA suite artifact dir conforming to
engine/schemas/tasks.schema.json, with "stack": "qa". Scenarios are independent, so each
scenario is its own single-task group.
Author it in one shot, then refine once. Compose the entire tasks.json — manifest, every
tasks[] entry, and slices[] — in a single Write. Do not stub the file and grow it with
successive Edits; assemble the whole structure in memory first and emit it once. After the
one Write, run the validator (below): if it prints OK you are done; if it fails, make one
corrective Edit (or a single rewriting Write) and re-validate. Repeat only to fix validation
errors, never to build the file up incrementally.
Fields:
context_manifest.read_once = shared fixtures / page objects / test helpers the specs use;
reference = the acceptance-criteria path and the contract summary.
- One
tasks[] entry per scenario: id, its own group_id, title (the journey), empty
depends_on, reads (extra fixtures), writes (the spec file it creates), test (the
scenario id), standards (e.g. ["risk-tiered","determinism","isolation"]),
needs_human_gate: false. slices[] = one group per scenario.
- Validate before returning:
python3 engine/validate_tasks.py <the tasks.json you wrote>
must print OK.
Scenario mode: when invoked with a group_id and tasks_path, author only that one
scenario's spec file (batch-load the fixture manifest once) and return spec_path.
Output
Write these to the artifact path(s) your instructions specify (the orchestrator passes them;
standalone, use a sensible dir you choose): a coverage note (journeys covered,
negatives covered, out-of-scope, data strategy), the run report, the tasks.json, and — when your
instructions ask for it — a per-case results file: every catalog case ID with passed, failed
or skipped and a short note for each failure or skip. Every catalog case appears; a case you did
not automate is skipped with the reason, never left out or reported as passed.
Definition of done
Critical journeys + key negatives automated and green in a clean env; tests isolated and
deterministic; failures produce artifacts; nothing weakened to pass.
Output contract
Return report_path, passed (true/false — did the whole suite pass), failed_count
(integer), and summary (one line: totals and the headline failure, if any) — all short
scalars.
1---2name: qa-automation3description: Author critical-journey E2E / UI-automation tests from the acceptance criteria (not from the implementation) in the project's own E2E framework, run them in a clean env, and report; a project without one gets each case checked directly against the running system, never a new framework. Meets QA standards (risk-tiering, isolation, determinism, no weakened assertions). Edits test files only. Front door for /qa-automation.4allowed-tools: Read, Grep, Glob, Bash, Edit, Write5tags: [sdlc, qa]6metadata:7 source: https://skillmd.com/skills/keyvaluesoftwaresystems/qa-automation8---910# qa-automation1112Author end-to-end / UI-automation tests derived from the **acceptance criteria**, not from13the implementation. Cover risk-tier **critical journeys** plus their key negative paths —14not every path.1516Test against observable behavior, not the implementation's internals, and **never weaken17assertions** to make a test pass.1819## Inputs20Your instructions name what to read — the acceptance criteria, the functional test-case21catalog, and (for cross-service journeys) the contract — and the artifact path to write.22Standalone? read them and write to a path you choose (and tell the user where).2324## Steps251. **Use the project's E2E framework, never a new one.** When your instructions name the26 project's E2E suite (or, standalone, the project already has one: e.g. Playwright, Cypress,27 WebdriverIO, Maestro for mobile), automate in that framework, reusing its config, fixtures and28 helpers, without installing packages or browsers. When the project has no E2E framework, do29 not add one and write no suite: check each case directly against the running system with the30 tools already there (HTTP requests, the CLI, database queries, the repositories' own test31 runners), record the observed result as the case's evidence, and skip the `tasks.json` and32 scenario steps below. The standards below still apply to what you check and how you report.332. **Source the journeys.** If `test_cases_path` exists, that catalog is the source of truth —34 automate its cases, preserving each case's ID/traceability, and do not silently drop a case35 (note any you defer and why). Only when no catalog is present do you derive journeys straight36 from the acceptance criteria. Either way: rank by risk and scope to the default tier —37 **critical-journeys-only**; pick the critical38 ones and the highest-value negative paths; state what is out of scope.393. **Design test data** — each test seeds and tears down its own data; no shared mutable state.404. **Author tests** with stable selectors (roles/test-ids), explicit waits on conditions,41 and assertions that verify observable behavior + the contract's outcomes.425. **Emit the scenario DAG** — write a `tasks.json` (see section below) so the43 suite can be authored in parallel.446. **Run in a clean env** — when your instructions say the environment is already running45 (the caller started it from the code under test), seed → run against it, and leave starting46 and stopping to the caller. Otherwise `stack up` → seed → ready_check → run → `stack down`47 (teardown always runs).487. **On failure**, apply the fix-loop rule (selectors/flakes/waits only) — one attempt per49 invocation; the workflow's `max_visits` on the fix node (typically 3) bounds the loop.5051## Standards every suite must satisfy52- **Risk-tiered** — critical journeys + key negatives; explicit out-of-scope note.53- **Isolated & idempotent** — self-contained setup/teardown; order-independent.54- **Deterministic** — wait on conditions/selectors, never fixed sleeps; stable selectors,55 not brittle CSS chains; control time/randomness where possible.56- **Cross-service** — real contract, not mocks, in the clean stack for cross-service journeys.57- **No weakened assertions** — self-healing may refresh selectors / re-run only; it must NOT58 weaken assertions, permissions, or expected behavior. Approved tests are frozen.59- **Safe** — no real secrets or production data; seeded test data only.60- **Reportable** — on failure, capture screenshot/trace/video and the failing step.6162## Edge cases / journeys to include63- Primary happy journey end-to-end across services.64- Validation failures surfaced to the user; permission-denied journey.65- Empty state and "many items" / pagination boundary.66- Slow/failed API handling (retry/error UI); session expiry mid-flow.67- Concurrency/double-submit where the journey allows it.68- Mobile: gestures, back button, deep links, orientation (Maestro).6970## External skill (provision — test generation)71If a suitable test-generator skill is installed you may delegate test generation to it —72the output must still meet the standards above (isolation, determinism, real assertions,73risk-tiering). Otherwise author in-pack.7475## Emit tasks.json (parallel scenario authoring)76Write a `tasks.json` under your QA suite artifact dir conforming to77`engine/schemas/tasks.schema.json`, with `"stack": "qa"`. Scenarios are independent, so each78scenario is its **own single-task group**.7980**Author it in one shot, then refine once.** Compose the entire tasks.json — manifest, every81`tasks[]` entry, and `slices[]` — in a single `Write`. Do **not** stub the file and grow it with82successive `Edit`s; assemble the whole structure in memory first and emit it once. After the83one `Write`, run the validator (below): if it prints `OK` you are done; if it fails, make **one**84corrective `Edit` (or a single rewriting `Write`) and re-validate. Repeat only to fix validation85errors, never to build the file up incrementally.8687Fields:88- `context_manifest.read_once` = shared fixtures / page objects / test helpers the specs use;89 `reference` = the acceptance-criteria path and the contract summary.90- One `tasks[]` entry per scenario: `id`, its own `group_id`, `title` (the journey), empty91 `depends_on`, `reads` (extra fixtures), `writes` (the spec file it creates), `test` (the92 scenario id), `standards` (e.g. `["risk-tiered","determinism","isolation"]`),93 `needs_human_gate: false`. `slices[]` = one group per scenario.94- **Validate before returning:** `python3 engine/validate_tasks.py <the tasks.json you wrote>`95 must print `OK`.9697**Scenario mode:** when invoked with a `group_id` and `tasks_path`, author only that one98scenario's spec file (batch-load the fixture manifest once) and return `spec_path`.99100## Output101Write these to the artifact path(s) your instructions specify (the orchestrator passes them;102standalone, use a sensible dir you choose): a coverage note (journeys covered,103negatives covered, out-of-scope, data strategy), the run report, the `tasks.json`, and — when your104instructions ask for it — a per-case results file: every catalog case ID with `passed`, `failed`105or `skipped` and a short note for each failure or skip. Every catalog case appears; a case you did106not automate is `skipped` with the reason, never left out or reported as passed.107108## Definition of done109Critical journeys + key negatives automated and green in a clean env; tests isolated and110deterministic; failures produce artifacts; nothing weakened to pass.111112## Output contract113Return `report_path`, `passed` (true/false — did the whole suite pass), `failed_count`114(integer), and `summary` (one line: totals and the headline failure, if any) — all short115scalars.