Testing & QA
Balance: ~40% Backend/API, ~40% Frontend/E2E (Playwright), ~20% Cross-cutting (CI, Security, TDD, Coverage) -- directional guidance, not a hard ratio.
Scope: test harnesses, fixtures, mock policy, runners, Playwright, CI gates, coverage. Root-cause analysis and debugging playbooks → jaw-dev-debugging.
- This skill owns test adequacy;
jaw-dev-code-reviewer owns finding severity and review process.
- CI pipeline ownership and deployment verification: see
jaw-dev-devops.
- Data pipeline testing and ETL validation: see
jaw-dev-data.
- Design direction context for rendered verification: see
jaw-dev-uiux-design.
This skill activates by change surface when work needs verification depth, regression coverage, or a reproducible test harness.
C0/C1 work (small local patches): See jaw-dev §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.
jaw-dev is canonical: jaw-dev §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill.
Modular References
| File |
When to Read |
What It Covers |
references/core/crud-test-matrix.md |
When choosing verification depth for a classified task, or testing a CRUD slice |
Risk-tier minimums, per-operation negatives, UI smoke rule |
references/manual-surface-qa.md |
Driving a changed API, CLI, TUI, web, or desktop surface |
Faithful channel, scenario matrix, verdict, teardown; desktop artifact criteria link to ../jaw-dev-devops/references/native-desktop-acceptance.md |
references/edge-first-testing.md |
New unit/service/integration tests for features (skip for regression/contract tests) |
Edge-first principle, test order by change type, 11-class edge matrix |
references/backend-testing.md |
Backend/API testing |
Supertest patterns, DB fixtures, auth mocking |
references/ci-pipeline.md |
CI configuration |
GitHub Actions, gates, caching, parallelism |
references/load-testing.md |
Performance/load testing, C3+ production readiness |
k6/Locust, test types, measure→profile→verify, CI gates |
references/ml-evaluation.md |
ML model/LLM evaluation, quality gates |
LLM-as-judge, RAGAS, DeepEval, CI eval gate, regression detection. CI eval gates are dataset-versioned + regression-based: pin prompts/models/retrieval config, preserve traces, calibrate judges on golden examples, fail only on meaningful regressions or safety failures |
When tests depend on current external API behavior, provider docs, CI service
behavior, test-environment versions, dependency audit evidence, or recorded
mock/fixture sources, read the active search skill and follow its
source-fetch and evidence-status rules.
1. Test Strategy
1.1 Models
| Model |
Best For |
Emphasis |
| Test Pyramid |
monoliths, libraries |
speed, isolation |
| Testing Trophy |
modern web apps, REST backends |
confidence-to-cost |
| Test Honeycomb |
microservices, async systems |
boundary verification |
1.2 Recommended Trophy Distribution
| Layer |
Default Share |
Typical Tools |
| Static analysis |
base layer |
tsc, ESLint, mypy, Ruff |
| Unit |
~25% |
Vitest, Jest, pytest |
| Integration |
~50% |
Supertest, httpx, Testcontainers |
| Contract |
~10% |
Pact, OpenAPI validators, Schemathesis |
| E2E |
~10% |
Playwright |
| Manual / exploratory |
~5% |
human review |
1.3 Risk-First Priorities
- auth / session / permission boundaries
- money movement, quota, credits
- data mutation and irreversible actions
- file upload / parsing / external webhooks
- shared API contracts used by frontend clients
- error paths, retries, rollback behavior
1.4 Harness Selector
| Problem |
Primary Harness |
Avoid |
| pure business rule |
unit / service test |
browser test |
| route + middleware + serialization |
API integration test |
mocking the route itself |
| DB query / migration / transaction |
real DB integration test |
fake repository for SQL correctness |
| frontend consuming backend JSON |
contract test |
manual-only verification |
| rendered critical flow |
Playwright smoke |
asserting internal React state |
| rendered artifact (visual correctness) |
render-grounding loop (jaw-dev-pabcd C-RENDER-GROUNDING-01) |
static parse / tsc alone |
1.5 General Rules
- Write tests for new features, bug fixes, refactors, and behavior changes.
- Prefer one behavioral concern per test.
- Use factories / builders for setup; avoid repeated inline blobs.
- A fast real dependency beats a mock. A mock beats an untested branch.
- If the failure is mysterious, delegate methodology to
jaw-dev-debugging, then return here for the regression harness.
- STRICT (TEST-ANTI-FLAKE-01): A time-based flake is a bug. Do not use sleep-based synchronization, retry-as-fix, or green-on-retry acceptance without a deterministic cause and harness correction.
- Verification depth follows
jaw-dev §3 DEV-VERIFY-FLOOR-01; CRUD per-operation negative coverage is owned by references/core/crud-test-matrix.md.
1.5a Limited-Oracle / Score-Objective Evaluation
When the real evaluator is scarce, paid, rate-limited, or opaque and local tests are
proxy metrics for a score/objective, apply §9.5 (single owner of GATE-ORACLE-VALIDITY-01,
GATE-PREFIX-HORIZON-01, GATE-INVARIANT-EV-01, GATE-HOLDOUT-LEAKAGE-01,
GATE-AGREEMENT-STATS-01). Pairs with jaw-dev-pabcd §10 Optimization-Loop Meta-Rules.
1.6 Property-Based & Mutation Testing (verified 2026-07-02)
| Technique |
Use for |
Default tools |
When |
| Property-based |
Pure logic, parsers, serializers, state machines, API invariants |
fast-check (TS), Hypothesis (Python) |
DEFAULT for invariant-heavy code |
| Mutation |
Judging test-suite strength on critical domain logic, validators, security branches |
Stryker (JS/TS), mutmut (Python) |
Selective, after stable unit/property tests — not every PR |
- Vitest 4 is the current runner baseline: Browser Mode is stable (visual regression via
toMatchScreenshot, Playwright trace generation, expect.schemaMatching).
2. Backend & API Testing
Deep reference: references/backend-testing.md
2.1 Coverage Map
| Layer |
Verify |
TypeScript Default |
Python Default |
| Service layer |
validation, orchestration, domain errors |
Vitest |
pytest |
| API layer |
status, envelope, middleware, auth |
Supertest |
httpx / ASGITransport |
| Repository layer |
SQL / ORM correctness |
Testcontainers + real DB |
Testcontainers + real DB |
| Background jobs |
idempotency, retry, dead-letter |
Vitest + fake clock |
pytest + monkeypatch |
2.2 Mock Strategy Hierarchy
real deterministic dependency
→ Testcontainers / ephemeral infra
→ recorded responses / thin fake
→ manual stub / fake
→ framework mock as last resort
2.3 Service & API Patterns
Mock dependencies at service boundaries. Use Supertest/httpx for route-level integration tests. Match response envelope shape from backend contracts.
2.4 Database Truth with Testcontainers
Use a real database when verifying migrations, transactions, unique constraints, foreign keys, query translation, and performance-sensitive SQL. Use Testcontainers for real DB truth in correctness-sensitive persistence tests. Start container in beforeAll/fixture setup, capture connection URI.
2.5 Fixture / Seed Synchronization
- Prefer builders / factories over copied JSON snapshots.
- Keep shared contract examples in
fixtures/contracts/ or equivalent.
- Seed data should expose stable IDs used by Playwright smoke flows.
- If frontend mocks drift from backend fixtures, write or update a contract test first.
3. Contract Testing
Contract tests protect the frontend↔backend boundary. They sit between API tests and browser tests.
Rule: Playwright proves the experience. Contract tests prove the shared shape.
3.1 Contract-Stable Surface
- response envelope:
success, data, error, meta
- error taxonomy: HTTP status + machine-readable
error.code
- pagination fields, auth headers, cookie behavior
requestId propagation
- nullability, timestamps, enums, money serialization
3.2 Contract Options
| Style |
Best For |
Tooling |
| consumer-driven contract |
rapidly changing frontend/backend teams |
Pact |
| schema-first contract |
OpenAPI-led backends |
OpenAPI validators, Schemathesis |
| type-level contract |
TS monorepos |
shared types / codegen |
| full-stack smoke |
final user confidence |
Playwright |
3.3 Consumer Contract — TypeScript (Pact / PactV4)
PactV4 (aliased Pact) is the current interface and supports Pact Specification v4;
treat PactV3 as the legacy spec-v3 API. Workflow:
- Define interaction: provider state + request + expected response (matchers for flexible matching)
- Execute test against Pact mock server
- Assert consumer expectations
- Pact file auto-writes to
pacts/ → publish to broker → provider verifies
See references/backend-testing.md for a full example.
3.4 Schema Verification
Use schema-based API testing (Schemathesis) to verify OpenAPI/GraphQL contract
compliance. (Dredd is legacy/inactive — do not adopt for new projects.)
3.5 Rules
- Contract tests are strongly recommended for parallel FE/BE, public APIs, and cross-team contracts.
- E2E success does not replace provider verification.
- Store golden examples near the contract, not inside one app only.
- If the shape is intentionally breaking, update the contract first, then all consumers.
4. Playwright Browser Testing
Use Playwright after API and contract tests are already trustworthy. Browser tests should validate rendered flows, accessibility-critical interactions, and real integration seams that lower layers cannot prove alone.
Helper Scripts Available:
scripts/with_server.py - Manages server lifecycle (supports multiple servers)
Run scripts with --help first — treat as black boxes to avoid context window pollution.
4.1 Decision Tree: Choosing Your Approach
User task → Static HTML? → Read file → find selectors → write Playwright script
→ Dynamic app? → Server running? → No: `python scripts/with_server.py --help`
→ Yes: Recon-then-action (navigate → screenshot → selectors → act)
4.2 Example: Using with_server.py
# Single server:
python scripts/with_server.py --server "npm run dev" --port 5173 -- python your_automation.py
# Multiple servers:
python scripts/with_server.py \
--server "cd backend && python server.py" --port 3000 \
--server "cd frontend && npm run dev" --port 5173 \
-- python your_automation.py
4.3 Reconnaissance-Then-Action Pattern
- Wait for an explicit app-ready signal or locator assertion → 2. Screenshot/inspect DOM → 3. Identify selectors → 4. Execute actions
4.4 Best Practices
- Use bundled scripts as black boxes — run
--help first, invoke directly.
- Use
sync_playwright() for synchronous scripts; always close the browser.
- Prefer locator-based interactions and web-first assertions:
expect(page.get_by_role("button", name="Save")).to_be_visible(), then click() on that locator.
- Prefer user-facing locators, especially
get_by_role() with an accessible name. Use get_by_label(), get_by_placeholder(), or get_by_test_id() when role/name cannot express the target.
- Avoid
networkidle, hard sleeps, and wait_for_timeout() in tests. Wait on observable app-ready signals, locator actions, or expect() assertions.
- AI-authored tests (DEFAULT): Playwright MCP / Test Agents (planner/generator/healer) are generation-and-repair aids only — final acceptance still requires deterministic locators, web-first assertions, traces, and a human-readable failure artifact.
4.5 Reference Files
- examples/ - Examples showing common patterns:
element_discovery.py - Discovering buttons, links, and inputs on a page
static_html_automation.py - Using file:// URLs for local HTML
console_logging.py - Capturing console logs during automation
4.6 Browser Testing Rules
- Run contract tests and API tests first for broken-data bugs.
- Use Playwright for rendered truth, not as a replacement for service tests.
- Prefer one smoke flow per critical path over many brittle micro-flows.
- If a failure looks like data-shape drift, go back to §2 Backend & API Testing or §3 Contract Testing.
4.7 Native Computer-Use / Browse-Use QA (exploratory tier) (TEST-CU-QA-01)
For rendered implementation context, load jaw-dev-frontend. Playwright owns
deterministic suites; exploratory QA follows ../jaw-dev/references/browse-qa-ladders.md
(QA-TOOL-LADDER-01). Use inspect → act → re-inspect. Jaw browser supports
cli-jaw browser snapshot --interactive, click, screenshot, text, and
get-dom; read captured screenshots. Missing access is a verification gap,
not a pass. Promote flows needing durable coverage to Playwright.
5. CI Pipeline Integration
Full workflow templates: references/ci-pipeline.md
5.1 Pipeline Order
quality (lint / typecheck)
→ unit + integration tests
→ contract tests
→ Playwright E2E
→ security scan
→ coverage aggregation + artifacts
5.2 Pipeline Template
Structure CI jobs in dependency chain: quality → backend-tests → contract-tests → e2e
Key configuration:
concurrency.cancel-in-progress: true — avoid wasted runs
strategy.fail-fast: false — for matrix builds
- Shard large suites:
--shard=${{ matrix.shard }}/N
- Install Playwright deps:
npx playwright install --with-deps chromium
See references/ci-pipeline.md for full GitHub Actions and GitLab CI templates.
5.3 Matrix & Parallelization
| Dimension |
When to Use |
| Node / Python version matrix |
packages, SDKs, shared libraries |
| OS matrix |
native modules, CLI behavior |
| shard matrix |
large suites exceeding CI budget |
npx vitest run --shard=1/4
npx playwright test --shard=1/4 --workers=4
pytest -n auto --dist=loadgroup
5.4 Flaky Test Remediation
| Symptom |
First Fix |
| passes locally, fails in CI |
deterministic seeds, containerized deps, explicit waits |
| order-dependent failure |
reset shared state in fixtures |
| green on retry only |
remove wall-clock / random assumptions |
| screenshot noise |
stable CI image, mask dynamic regions |
| Protocol: detect → quarantine if blocking → assign owner → reinstate after repeated green runs. |
|
5.5 Rules
- Do not let Playwright be the only blocking job.
- Contract tests should run before browser tests.
- Upload artifacts for failures: coverage, junit, traces, screenshots.
- Fail the build on broken thresholds, not only test exit codes.
6. TDD Enforcement Mode
When ENFORCE_TDD=true is set in project instructions or explicitly requested, this section becomes mandatory.
6.1 RED → GREEN → REFACTOR
- RED — write the failing test first and verify it fails for the right reason.
- GREEN — write the minimum implementation to pass.
- REFACTOR — clean up after green, then rerun the affected suite.
6.2 Self-Audit Checklist
| Check |
Pass Criteria |
| Test written before implementation? |
test file added / updated before or with code |
| Failure observed before fix? |
red state was actually executed |
| Behavior-focused assertions? |
checks outputs, side effects, contracts |
| Regression locked in? |
failing case is now protected by a persistent test |
6.3 Vertical Tracer-Bullet TDD
Prefer one behavior test → minimal implementation → next behavior. Slice by something
a user, caller, or consuming module can observe, not by horizontal layers such as "DB",
"API", then "UI". Assert through public interfaces and durable contracts. Retire shallow
scaffolding tests when a stronger interface or acceptance test covers the same promise.
6.4 Default Style
| Style |
Best For |
| London / mockist |
orchestration-heavy boundaries |
| Chicago / classicist |
domain logic and transforms |
| Hybrid |
most production code |
| Default to Hybrid: mock external systems, keep internal collaboration real unless it becomes too slow or unstable. |
|
6.5 Boundary with jaw-dev-debugging
jaw-dev-testing owns the regression harness and enforcement loop.
jaw-dev-debugging owns root-cause methodology once a failure is mysterious or multi-layered.
- After
jaw-dev-debugging isolates the cause, come back here to lock it in with tests.
6.6 AI-Assisted Development Regressions
When an AI writes and reviews its own code, it carries the same assumptions into both steps. Automated tests break this feedback loop.
Common AI Regression Patterns
| Pattern |
Description |
Test Strategy |
| Sandbox/production mismatch |
Fix applied to one code path, not both |
Assert same response shape in both modes |
| SELECT clause omission |
New field in response but missing from DB query |
Assert all required fields are present and defined |
| Error state leakage |
Error set but stale data not cleared |
Assert state cleanup on error transitions |
| Missing rollback |
Optimistic UI update without recovery on failure |
Assert state restoration after simulated API error |
Regression Naming Convention
Name regression tests with BUG-R{N} convention. Assert all required fields with a loop.
Sandbox-Mode API Testing
When the project supports a sandbox/mock mode, use it for fast DB-free regression testing:
- Force sandbox mode in test setup:
process.env.SANDBOX_MODE = 'true'
- Assert sandbox responses match the same contract as production responses.
- Treat sandbox/production parity as a high-priority regression target.
- In sandbox/spike mode, write tests for bugs found — coverage grows organically. For production refactors, see §1.5 (tests required for behavior changes).
6.7 Test-Induced Production Defense Detection
Rule: Do not add production defensive code solely to satisfy unrealistic tests. A production guard is allowed only when the invalid state can occur at a real boundary or represents an explicit domain rule.
| Production change smell |
Likely test problem |
Required action |
Internal if (!x) return added after unit test fails |
Test fixture omitted required field |
Fix fixture factory or test boundary validation |
| Required field made optional to satisfy test |
Test is using invalid domain object |
Restore required type and update test data |
| Catch-all added so test passes |
Test expects silence instead of failure |
Assert typed error or user-visible failure |
| Production default added for impossible state |
Test bypassed constructor/parser |
Use real constructor/parser in test |
| Private helper exported only for test |
Test is coupled to implementation |
Test public behavior or move helper to test support |
| Sleep/retry added only for test flake |
Test lacks deterministic synchronization |
Wait on observable condition or fake clock |
NODE_ENV === "test" branch added |
Test-only production behavior |
Remove branch; improve test harness |
Required questions before adding a guard:
- Is the input from an untrusted boundary? → If yes, validate at that boundary
- Can this state happen in production? → If no, fix the test
- What contract allows this value? → Cite schema/type/domain rule
- Would this hide a real bug? → If yes, fail fast instead
Banned patterns: process.env.NODE_ENV === "test" branches, silent fallbacks for impossible internal state, making required types optional for mocks, exporting internals only for tests.
Allowed guards: Boundary validation (process/network/user/file boundary), backward compatibility (documented old schema), security checks, domain invariants, observed production bug regressions, external dependency adapters.
7. Accessibility Testing
Component Level
Page Level
CI Pipeline
- Gate order (verified 2026-07-02): component axe (jest-axe/vitest-axe) → page axe
(@axe-core/playwright) → keyboard/focus/manual checks. Blocking gate = zero
serious/critical axe violations + targeted manual checks.
- Lighthouse a11y score (≥90) is an advisory route-level smoke signal, NOT the blocking
gate — Lighthouse itself separates automated scoring from required manual checks.
- Pa11y: page-level scanning for WCAG AA violations
- Run a11y tests on EVERY page route, not just the homepage
Observability Verification
Verify trace propagation in integration tests. Assert that spans appear for critical paths. Check structured log format matches the schema in ../jaw-dev-backend/references/core/observability.md.
8. Security Testing
→ Delegated: threat modeling and secure design policy belong to jaw-dev-security.
This section covers the automated test hooks and CI gates that enforce those rules.
8.1 Minimum Security Stack
fast local checks
→ Semgrep / CodeQL gate
→ dependency audit
→ auth / validation regression tests
8.2 Dependency Scanning Commands
npm audit --audit-level=high
pip-audit --strict --desc
8.3 Semgrep Gate
The returntocorp/semgrep-action wrapper is deprecated (stated by the repo itself) —
run Semgrep natively in CI:
semgrep:
runs-on: ubuntu-latest
container: semgrep/semgrep
steps:
- uses: actions/checkout@v4
- run: semgrep ci --config p/default --config p/javascript --config p/typescript --config p/python
(Open-source alternative engine: Opengrep, the LGPL-2.1 community fork — see
../jaw-dev-security/references/static-analysis.md.)
8.4 Security Regressions
Test missing auth (expect 401) and verify error.code matches contract for every auth-protected endpoint.
8.5 Rules
- dependency audit in CI
- Semgrep or equivalent SAST
- auth / permission regression tests
- validation tests for malicious or malformed input
- a blocking rule for high / critical dependency findings
9. Coverage & Quality Gates
9.1 Suggested Thresholds
These are project/risk-based, not universal minimums. Adjust for your context.
| Metric |
Suggested Floor |
Ideal |
| Line coverage |
70% |
85%+ |
| Branch coverage |
60% |
80%+ |
| Function coverage |
80% |
90%+ |
| Diff coverage |
80% |
90%+ |
9.2 Outcome Metrics
| Metric |
Target |
| Defect detection rate |
> 80% |
| Mean time to detect |
< 1 CI run |
| Test signal-to-noise |
> 95% |
| Contract drift rate |
near 0 |
9.3 Coverage Workflow
- generate coverage reports
npm test -- --coverage
npx vitest run --coverage
pytest --cov --cov-report=xml
- review by priority: auth, payment, mutations, upload, contracts first
- write targeted tests for the gaps
- publish artifacts and fail the merge when thresholds drop
9.4 Quality Gate Checklist
9.5 Limited-Oracle / Score-Objective Evaluation
Use these rules when the true evaluator is scarce, paid, rate-limited, or opaque and
local checks are only proxy metrics for a score/objective. PABCD loop response to
repeated candidate deaths is owned by jaw-dev-pabcd §10 Optimization-Loop Meta-Rules.
- STRICT (GATE-ORACLE-VALIDITY-01): When the true evaluator/oracle is rate-limited
(limited submissions, paid runs) and local metrics are proxies, evaluator validity is a
prerequisite gate. Before trusting the proxy for accept/reject, quantify historical
divergence: cases where the proxy said better/equal but the oracle said worse. A proxy
with known optimistic bias must not be the sole acceptance evidence.
- DEFAULT (GATE-PREFIX-HORIZON-01): Replay-based evidence (recorded logs, scripted
opponents) is prefix-valid only; it stops being valid when the candidate diverges from
the recorded trajectory. Candidates that diverge early need live adversarial evaluation
through a modeled opponent/environment, not replays. State the divergence turn/point
when citing replay evidence.
- DEFAULT (GATE-INVARIANT-EV-01): Every hard invariant in an acceptance gate (a metric
that must not regress) needs an expected-value justification: protected value versus the
candidate-space it vetoes. If a hard invariant vetoes 3+ consecutive candidates that
target strictly larger gains, downgrade it to a soft cost and re-justify or remove.
- DEFAULT (GATE-HOLDOUT-LEAKAGE-01): Fixed evaluation sets become training data
under adaptive reuse. Rotate or quarantine repeatedly used sets, keep a blind final
set, and treat repeated score-driven tuning on the same instances as leakage risk.
Grounding: Blum and Hardt, "The Ladder: A Reliable Leaderboard for Machine
Learning Competitions" (arXiv:1502.04585), and Dwork et al., "The reusable holdout:
Preserving validity in adaptive data analysis" (arXiv:1506.02629).
- HEURISTIC (GATE-AGREEMENT-STATS-01): For proxy-vs-oracle validation, report
agreement statistics that expose error shape: sign-discordance, bias, and worst-case
error. Correlation alone is not enough; Bland-Altman style agreement analysis is the
grounding model.
Grounding: observed in a 14-discard optimization plateau where a prefix-only replay gate
and a hard draw-protection invariant locked a 3.5/8 score.
10. Pre-Flight Test Checklist
10.1 Change-Type Routing
10.2 Harness Readiness
10.3 Contract & Data Integrity
10.4 CI & Reporting
10.5 Final Rule (risk-tier)
Verification intensity follows the work class (jaw-dev §0.0 / references/core/crud-test-matrix.md):
for C2 UI work, one focused smoke (manual click-through or one Playwright run) plus targeted
checks IS a complete story; for C3/C4 or release-sensitive work, a single smoke is not enough —
run the affected suites and required negatives. Manual/Playwright smoke is a risk-tier rule,
not a universal blocker.
unit / service
→ API integration
→ contract verification
→ Playwright smoke
→ CI gate + coverage + security scan
Patch Integrity Gate (TEST-PATCH-INTEGRITY-01, DEFAULT)
Source: sol research (SWE-bench containerized evaluation, addyosmani/agent-skills).
An agent that obtains green by weakening tests has not fixed the bug. Before
claiming implementation complete:
- Baseline: record which tests fail and their failure signatures BEFORE any
production code change.
- Implement: write the fix/feature.
- Re-run originals: execute the ORIGINAL test suite (not the modified version)
against the patched source. All baseline failures must now pass.
- Classify test changes: every test/config change is
required (new test for
new behavior), suspicious (deleted assertion, lowered threshold, added skip,
reduced coverage exclusion), or unrelated.
- Justify suspicious changes: each suspicious change needs a stated reason.
"The test was wrong" is valid only with evidence of the original test's incorrectness.
Red flags that trigger escalation:
- Deleted assertions without replacement
- Snapshot updates without visual/behavioral verification
- Coverage exclusions added in the same PR as the fix
@skip or .skip() added to failing tests
- Threshold reductions (e.g., coverage 80% → 60%)
- Type assertion suppressions (
as any, @ts-ignore) in test files
TDD Evidence Contract (TEST-TDD-EVIDENCE-01, DEFAULT)
When TDD is claimed, durable evidence must show:
- RED: failing test name + failure message (before production code)
- GREEN: same test passing (after production code)
- REFACTOR: full affected suite passing (after cleanup)
A TDD claim without RED evidence is not TDD.
Acceptance-Row Reachability (TEST-ROW-REACHABLE-01, DEFAULT)
Every row of an acceptance-criteria table must have a constructible precondition.
Before writing a row, ask: is there a call path that reaches this state? Does an
earlier guard consume this condition first, so the branch under test is never entered?
Does an operation that produces this value actually exist?
An unreachable row is decoration, not verification — the implementer tries to write
that test, cannot, and quietly drops it. This applies C-ACTIVATION-GROUNDING-01's
requirement (every conditional path names its activation scenario) to each row of the
acceptance table, not just to the plan's prose.
Common unreachable shapes:
- The row asserts a rejection that an earlier, broader rule already rejects — the
specific guard is never exercised, so the test passes even if that guard is absent.
- The row needs a state the public API cannot produce (no operation creates it).
- The row asserts on a call-site argument the function never receives.
- Two rows in the same table are mutually exclusive on the same tree (for example
"the gate reports a failure here" together with "the gate exits 0 overall").
Test Oracle Integrity (DEFAULT)
Three narrow contracts that stop false-green. No gate enforces any of them; the
reviewer is the only check.
TEST-PROMPT-SEAM-01. Do not assert on prose. A test may read a document only when
it extracts a value and compares it against a value from another source. Asserting
that a phrase exists in a file is a violation no matter how many files you read or
what the test's header says it is for.
- Forbidden:
assert.match(readFileSync(".../SKILL.md"), /prefer rg first/i) — one
source, phrase existence, breaks on harmless rewording, proves no behavior.
- Allowed: parse the frontmatter
description, pull a token out of it, and assert that
token appears in the runtime constant it is supposed to mirror — two sources, values
compared, breaks only when they genuinely disagree.
- Outside this rule's scope, and allowed: asserting on non-prose values — version pins,
license names, runtime output, CLI stdout, file existence.
The rule was written after two such tests were deleted rather than repaired, and
the reason repair failed is the reusable part. Three designs were tried: a structured
metadata field in each document compared between them (nothing at runtime reads such a
field, so it becomes a third source of truth that passes whenever both copies are wrong
together); promoting the prose to a behavioral test (the behavior was already covered,
so this duplicated coverage while appearing to offset the deletion); and promoting a
doctrine statement the same way (no runtime branch implements it, so there is nothing to
compare against).
What that costs is stated plainly rather than hidden: the affected doctrine has no
automated consistency check and is a human-review item. When prose has no
counterpart in code, a test that reads it can only check that the words are still
there. Deleting it removes a false green; inventing a second document to compare it
against removes nothing and adds a lie.
TEST-ORACLE-INDEPENDENCE-01. Never derive the expected value from the code under
test.
- Forbidden:
assert.equal(fn(x), fn(x)); building the expectation with a helper the
code under test also uses; refreshing a snapshot from current output and calling that
verification.
- Allowed: hardcode the expectation in the fixture, or compute it by an independent
route — a second implementation, a hand-worked example, an external spec.
TEST-PRECEDENCE-FIXTURE-01. When testing override / default / fallback, the three
values must all differ — otherwise the test cannot tell which path ran.
- Forbidden: override
"x", default "x", fallback "x" — every branch passes.
- Allowed: override
"from-flag", default "from-config", fallback "builtin", with
each case asserting the specific one.
A regression test should FAIL when the defect is reintroduced. Confirm it once by
mutation — break it, watch it go red, restore it, watch it go green. A regression test
that has never been seen red is an assertion about the test, not about the defect.
CI Green Discipline (TEST-CI-GREEN-01, STRICT)
The latest HEAD is the source of truth for whether CI is green.
- Inspect before editing. Read the failing job's log and its artifacts. A failure
you have not read is a failure you are guessing about.
- Make the minimal correct fix for that failure.
- Re-watch the latest HEAD, not the run you were looking at when you started.
Never blind-retry a failed job, and never push another change without new failure
evidence. Both produce the same illusion — activity that looks like progress while the
failure is unexamined. Re-running to measure a failure RATE is a different act and is
allowed; see references/ci-pipeline.md §5.2.
Exploratory QA Tier (TEST-CU-QA-01, DEFAULT)
Deterministic browser test suites and exploratory QA are different tools for different
jobs, and the split is worth stating because collapsing it wastes both:
- Deterministic suites own regression guarding. They are the thing you keep.
- Exploratory QA through whatever browser or screen-control capability the runtime
offers owns immediate proof that a surface built right now actually works.
Protocol: inspect → act → re-inspect. When DOM inspection is unavailable, fall back
to screenshots and read the image back — a screenshot produced but never read is
not an observation.
Evidence names the flow, the states traversed, the result, and the artifacts. Promote
any flow that must stay guarded into a deterministic test; exploratory proof does not
accumulate, so a flow left only in QA is a flow that will regress unnoticed.
The tool ladder for this tier is QA-TOOL-LADDER-01 in
../jaw-dev/references/browse-qa-ladders.md; use an available faithful
capability and state why a ladder step was skipped.
1---2name: jaw-dev-testing3description: "MUST USE for testing, QA, regression protection, and release verification — unit, integration, API, contract, Playwright E2E, CI, security-scan, coverage, and TDD strategy. Triggers: write tests, regression test, Playwright, E2E, contract test, coverage, CI flake, TDD, test, testing, QA, 테스트, 회귀 테스트, 품질 게이트."4metadata:5 source: https://skillmd.com/skills/lidge-jun/jaw-dev-testing6 short-description: "Testing and QA router: strategy, harness choice, CI gates, TDD, and coverage."7 keywords: "test, testing, TDD, coverage, regression, e2e, playwright, contract test, CI, property-based, mutation"8 last-verified: "2026-07-02"9---1011# Testing & QA12Balance: ~40% Backend/API, ~40% Frontend/E2E (Playwright), ~20% Cross-cutting (CI, Security, TDD, Coverage) -- directional guidance, not a hard ratio.13**Scope**: test harnesses, fixtures, mock policy, runners, Playwright, CI gates, coverage. Root-cause analysis and debugging playbooks → `jaw-dev-debugging`.14- This skill owns test adequacy; `jaw-dev-code-reviewer` owns finding severity and review process.15- CI pipeline ownership and deployment verification: see `jaw-dev-devops`.16- Data pipeline testing and ETL validation: see `jaw-dev-data`.17- Design direction context for rendered verification: see `jaw-dev-uiux-design`.18This skill activates by change surface when work needs verification depth, regression coverage, or a reproducible test harness.1920> **C0/C1 work (small local patches):** See `jaw-dev` §0.0 Work Classifier + §0.1 Patch Fast-Path before reading references.2122> **`jaw-dev` is canonical:** `jaw-dev` §0.2 Rule Classes, §3 Verification Gate, and §5 Safety Rules apply to all work governed by this skill.2324## Modular References2526| File | When to Read | What It Covers |27|------|-------------|----------------|28| `references/core/crud-test-matrix.md` | When choosing verification depth for a classified task, or testing a CRUD slice | Risk-tier minimums, per-operation negatives, UI smoke rule |29| `references/manual-surface-qa.md` | Driving a changed API, CLI, TUI, web, or desktop surface | Faithful channel, scenario matrix, verdict, teardown; desktop artifact criteria link to `../jaw-dev-devops/references/native-desktop-acceptance.md` |30| `references/edge-first-testing.md` | New unit/service/integration tests for features (skip for regression/contract tests) | Edge-first principle, test order by change type, 11-class edge matrix |31| `references/backend-testing.md` | Backend/API testing | Supertest patterns, DB fixtures, auth mocking |32| `references/ci-pipeline.md` | CI configuration | GitHub Actions, gates, caching, parallelism |33| `references/load-testing.md` | Performance/load testing, C3+ production readiness | k6/Locust, test types, measure→profile→verify, CI gates |34| `references/ml-evaluation.md` | ML model/LLM evaluation, quality gates | LLM-as-judge, RAGAS, DeepEval, CI eval gate, regression detection. CI eval gates are dataset-versioned + regression-based: pin prompts/models/retrieval config, preserve traces, calibrate judges on golden examples, fail only on meaningful regressions or safety failures |3536When tests depend on current external API behavior, provider docs, CI service37behavior, test-environment versions, dependency audit evidence, or recorded38mock/fixture sources, read the active `search` skill and follow its39source-fetch and evidence-status rules.4041---42## 1. Test Strategy43### 1.1 Models44| Model | Best For | Emphasis |45|-------|----------|----------|46| Test Pyramid | monoliths, libraries | speed, isolation |47| **Testing Trophy** | modern web apps, REST backends | confidence-to-cost |48| Test Honeycomb | microservices, async systems | boundary verification |4950### 1.2 Recommended Trophy Distribution51| Layer | Default Share | Typical Tools |52|-------|---------------|---------------|53| Static analysis | base layer | `tsc`, ESLint, mypy, Ruff |54| Unit | ~25% | Vitest, Jest, pytest |55| **Integration** | **~50%** | Supertest, httpx, Testcontainers |56| Contract | ~10% | Pact, OpenAPI validators, Schemathesis |57| E2E | ~10% | Playwright |58| Manual / exploratory | ~5% | human review |59### 1.3 Risk-First Priorities601. auth / session / permission boundaries612. money movement, quota, credits623. data mutation and irreversible actions634. file upload / parsing / external webhooks645. shared API contracts used by frontend clients656. error paths, retries, rollback behavior66### 1.4 Harness Selector67| Problem | Primary Harness | Avoid |68|---------|-----------------|-------|69| pure business rule | unit / service test | browser test |70| route + middleware + serialization | API integration test | mocking the route itself |71| DB query / migration / transaction | real DB integration test | fake repository for SQL correctness |72| frontend consuming backend JSON | contract test | manual-only verification |73| rendered critical flow | Playwright smoke | asserting internal React state |74| rendered artifact (visual correctness) | render-grounding loop (jaw-dev-pabcd C-RENDER-GROUNDING-01) | static parse / tsc alone |75### 1.5 General Rules76- Write tests for **new features, bug fixes, refactors, and behavior changes**.77- Prefer **one behavioral concern per test**.78- Use factories / builders for setup; avoid repeated inline blobs.79- A fast real dependency beats a mock. A mock beats an untested branch.80- If the failure is mysterious, **delegate methodology to `jaw-dev-debugging`**, then return here for the regression harness.81- **STRICT (TEST-ANTI-FLAKE-01):** A time-based flake is a bug. Do not use sleep-based synchronization, retry-as-fix, or green-on-retry acceptance without a deterministic cause and harness correction.82- Verification depth follows `jaw-dev` §3 `DEV-VERIFY-FLOOR-01`; CRUD per-operation negative coverage is owned by `references/core/crud-test-matrix.md`.8384### 1.5a Limited-Oracle / Score-Objective Evaluation8586When the real evaluator is scarce, paid, rate-limited, or opaque and local tests are87proxy metrics for a score/objective, apply §9.5 (single owner of GATE-ORACLE-VALIDITY-01,88GATE-PREFIX-HORIZON-01, GATE-INVARIANT-EV-01, GATE-HOLDOUT-LEAKAGE-01,89GATE-AGREEMENT-STATS-01). Pairs with `jaw-dev-pabcd` §10 Optimization-Loop Meta-Rules.9091### 1.6 Property-Based & Mutation Testing (verified 2026-07-02)9293| Technique | Use for | Default tools | When |94|-----------|---------|---------------|------|95| Property-based | Pure logic, parsers, serializers, state machines, API invariants | fast-check (TS), Hypothesis (Python) | DEFAULT for invariant-heavy code |96| Mutation | Judging test-suite strength on critical domain logic, validators, security branches | Stryker (JS/TS), mutmut (Python) | Selective, after stable unit/property tests — not every PR |9798- Vitest 4 is the current runner baseline: Browser Mode is stable (visual regression via99 `toMatchScreenshot`, Playwright trace generation, `expect.schemaMatching`).100---101## 2. Backend & API Testing102> Deep reference: `references/backend-testing.md`103### 2.1 Coverage Map104| Layer | Verify | TypeScript Default | Python Default |105|-------|--------|-------------------|----------------|106| Service layer | validation, orchestration, domain errors | Vitest | pytest |107| API layer | status, envelope, middleware, auth | Supertest | httpx / ASGITransport |108| Repository layer | SQL / ORM correctness | Testcontainers + real DB | Testcontainers + real DB |109| Background jobs | idempotency, retry, dead-letter | Vitest + fake clock | pytest + monkeypatch |110### 2.2 Mock Strategy Hierarchy111```text112real deterministic dependency113→ Testcontainers / ephemeral infra114→ recorded responses / thin fake115→ manual stub / fake116→ framework mock as last resort117```118### 2.3 Service & API Patterns119Mock dependencies at service boundaries. Use Supertest/httpx for route-level integration tests. Match response envelope shape from backend contracts.120### 2.4 Database Truth with Testcontainers121Use a **real database** when verifying migrations, transactions, unique constraints, foreign keys, query translation, and performance-sensitive SQL. Use Testcontainers for real DB truth in correctness-sensitive persistence tests. Start container in beforeAll/fixture setup, capture connection URI.122### 2.5 Fixture / Seed Synchronization123- Prefer builders / factories over copied JSON snapshots.124- Keep shared contract examples in `fixtures/contracts/` or equivalent.125- Seed data should expose **stable IDs** used by Playwright smoke flows.126- If frontend mocks drift from backend fixtures, write or update a **contract test first**.127---128## 3. Contract Testing129Contract tests protect the **frontend↔backend boundary**. They sit between API tests and browser tests.130**Rule**: Playwright proves the experience. Contract tests prove the shared shape.131### 3.1 Contract-Stable Surface132- response envelope: `success`, `data`, `error`, `meta`133- error taxonomy: HTTP status + machine-readable `error.code`134- pagination fields, auth headers, cookie behavior135- `requestId` propagation136- nullability, timestamps, enums, money serialization137### 3.2 Contract Options138| Style | Best For | Tooling |139|-------|----------|---------|140| consumer-driven contract | rapidly changing frontend/backend teams | Pact |141| schema-first contract | OpenAPI-led backends | OpenAPI validators, Schemathesis |142| type-level contract | TS monorepos | shared types / codegen |143| full-stack smoke | final user confidence | Playwright |144### 3.3 Consumer Contract — TypeScript (Pact / PactV4)145`PactV4` (aliased `Pact`) is the current interface and supports Pact Specification v4;146treat `PactV3` as the legacy spec-v3 API. Workflow:1471. Define interaction: provider state + request + expected response (matchers for flexible matching)1482. Execute test against Pact mock server1493. Assert consumer expectations1504. Pact file auto-writes to `pacts/` → publish to broker → provider verifies151152See `references/backend-testing.md` for a full example.153### 3.4 Schema Verification154Use schema-based API testing (Schemathesis) to verify OpenAPI/GraphQL contract155compliance. (Dredd is legacy/inactive — do not adopt for new projects.)156### 3.5 Rules157- Contract tests are **strongly recommended** for parallel FE/BE, public APIs, and cross-team contracts.158- E2E success does **not** replace provider verification.159- Store golden examples near the contract, not inside one app only.160- If the shape is intentionally breaking, update the contract first, then all consumers.161---162## 4. Playwright Browser Testing163Use Playwright after API and contract tests are already trustworthy. Browser tests should validate rendered flows, accessibility-critical interactions, and real integration seams that lower layers cannot prove alone.164**Helper Scripts Available**:165- `scripts/with_server.py` - Manages server lifecycle (supports multiple servers)166Run scripts with `--help` first — treat as black boxes to avoid context window pollution.167### 4.1 Decision Tree: Choosing Your Approach168```169User task → Static HTML? → Read file → find selectors → write Playwright script170 → Dynamic app? → Server running? → No: `python scripts/with_server.py --help`171 → Yes: Recon-then-action (navigate → screenshot → selectors → act)172```173### 4.2 Example: Using with_server.py174```bash175# Single server:176python scripts/with_server.py --server "npm run dev" --port 5173 -- python your_automation.py177178# Multiple servers:179python scripts/with_server.py \180 --server "cd backend && python server.py" --port 3000 \181 --server "cd frontend && npm run dev" --port 5173 \182 -- python your_automation.py183```184### 4.3 Reconnaissance-Then-Action Pattern1851. Wait for an explicit app-ready signal or locator assertion → 2. Screenshot/inspect DOM → 3. Identify selectors → 4. Execute actions186187### 4.4 Best Practices188- **Use bundled scripts as black boxes** — run `--help` first, invoke directly.189- Use `sync_playwright()` for synchronous scripts; always close the browser.190- Prefer locator-based interactions and web-first assertions: `expect(page.get_by_role("button", name="Save")).to_be_visible()`, then `click()` on that locator.191- Prefer user-facing locators, especially `get_by_role()` with an accessible name. Use `get_by_label()`, `get_by_placeholder()`, or `get_by_test_id()` when role/name cannot express the target.192- Avoid `networkidle`, hard sleeps, and `wait_for_timeout()` in tests. Wait on observable app-ready signals, locator actions, or `expect()` assertions.193- **AI-authored tests (DEFAULT):** Playwright MCP / Test Agents (planner/generator/healer) are generation-and-repair aids only — final acceptance still requires deterministic locators, web-first assertions, traces, and a human-readable failure artifact.194### 4.5 Reference Files195- **examples/** - Examples showing common patterns:196 - `element_discovery.py` - Discovering buttons, links, and inputs on a page197 - `static_html_automation.py` - Using file:// URLs for local HTML198 - `console_logging.py` - Capturing console logs during automation199### 4.6 Browser Testing Rules200- Run **contract tests and API tests first** for broken-data bugs.201- Use Playwright for **rendered truth**, not as a replacement for service tests.202- Prefer one smoke flow per critical path over many brittle micro-flows.203- If a failure looks like data-shape drift, go back to **§2 Backend & API Testing** or **§3 Contract Testing**.204205### 4.7 Native Computer-Use / Browse-Use QA (exploratory tier) (TEST-CU-QA-01)206For rendered implementation context, load `jaw-dev-frontend`. Playwright owns207deterministic suites; exploratory QA follows `../jaw-dev/references/browse-qa-ladders.md`208(QA-TOOL-LADDER-01). Use inspect → act → re-inspect. Jaw browser supports209`cli-jaw browser snapshot --interactive`, `click`, `screenshot`, `text`, and210`get-dom`; read captured screenshots. Missing access is a verification gap,211not a pass. Promote flows needing durable coverage to Playwright.212---213## 5. CI Pipeline Integration214> Full workflow templates: `references/ci-pipeline.md`215### 5.1 Pipeline Order216```text217quality (lint / typecheck)218→ unit + integration tests219→ contract tests220→ Playwright E2E221→ security scan222→ coverage aggregation + artifacts223```224### 5.2 Pipeline Template225Structure CI jobs in dependency chain: `quality → backend-tests → contract-tests → e2e`226227Key configuration:228- `concurrency.cancel-in-progress: true` — avoid wasted runs229- `strategy.fail-fast: false` — for matrix builds230- Shard large suites: `--shard=${{ matrix.shard }}/N`231- Install Playwright deps: `npx playwright install --with-deps chromium`232233See `references/ci-pipeline.md` for full GitHub Actions and GitLab CI templates.234### 5.3 Matrix & Parallelization235| Dimension | When to Use |236|-----------|-------------|237| Node / Python version matrix | packages, SDKs, shared libraries |238| OS matrix | native modules, CLI behavior |239| shard matrix | large suites exceeding CI budget |240```bash241npx vitest run --shard=1/4242npx playwright test --shard=1/4 --workers=4243pytest -n auto --dist=loadgroup244```245### 5.4 Flaky Test Remediation246| Symptom | First Fix |247|---------|-----------|248| passes locally, fails in CI | deterministic seeds, containerized deps, explicit waits |249| order-dependent failure | reset shared state in fixtures |250| green on retry only | remove wall-clock / random assumptions |251| screenshot noise | stable CI image, mask dynamic regions |252Protocol: detect → quarantine if blocking → assign owner → reinstate after repeated green runs.253### 5.5 Rules254- Do not let Playwright be the **only** blocking job.255- Contract tests should run **before** browser tests.256- Upload artifacts for failures: coverage, junit, traces, screenshots.257- Fail the build on broken thresholds, not only test exit codes.258---259## 6. TDD Enforcement Mode260When `ENFORCE_TDD=true` is set in project instructions or explicitly requested, this section becomes mandatory.261### 6.1 RED → GREEN → REFACTOR2621. **RED** — write the failing test first and verify it fails for the right reason.2632. **GREEN** — write the minimum implementation to pass.2643. **REFACTOR** — clean up after green, then rerun the affected suite.265### 6.2 Self-Audit Checklist266| Check | Pass Criteria |267|-------|--------------|268| Test written before implementation? | test file added / updated before or with code |269| Failure observed before fix? | red state was actually executed |270| Behavior-focused assertions? | checks outputs, side effects, contracts |271| Regression locked in? | failing case is now protected by a persistent test |272### 6.3 Vertical Tracer-Bullet TDD273Prefer one behavior test → minimal implementation → next behavior. Slice by something274a user, caller, or consuming module can observe, not by horizontal layers such as "DB",275"API", then "UI". Assert through public interfaces and durable contracts. Retire shallow276scaffolding tests when a stronger interface or acceptance test covers the same promise.277### 6.4 Default Style278| Style | Best For |279|-------|----------|280| London / mockist | orchestration-heavy boundaries |281| Chicago / classicist | domain logic and transforms |282| **Hybrid** | most production code |283Default to **Hybrid**: mock external systems, keep internal collaboration real unless it becomes too slow or unstable.284### 6.5 Boundary with jaw-dev-debugging285- `jaw-dev-testing` owns the **regression harness** and enforcement loop.286- `jaw-dev-debugging` owns **root-cause methodology** once a failure is mysterious or multi-layered.287- After `jaw-dev-debugging` isolates the cause, come back here to lock it in with tests.288---289## 6.6 AI-Assisted Development Regressions290291When an AI writes and reviews its own code, it carries the same assumptions into both steps. Automated tests break this feedback loop.292293### Common AI Regression Patterns294295| Pattern | Description | Test Strategy |296|---------|-------------|---------------|297| Sandbox/production mismatch | Fix applied to one code path, not both | Assert same response shape in both modes |298| SELECT clause omission | New field in response but missing from DB query | Assert all required fields are present and defined |299| Error state leakage | Error set but stale data not cleared | Assert state cleanup on error transitions |300| Missing rollback | Optimistic UI update without recovery on failure | Assert state restoration after simulated API error |301302### Regression Naming Convention303304Name regression tests with BUG-R{N} convention. Assert all required fields with a loop.305306### Sandbox-Mode API Testing307308When the project supports a sandbox/mock mode, use it for fast DB-free regression testing:309- Force sandbox mode in test setup: `process.env.SANDBOX_MODE = 'true'`310- Assert sandbox responses match the same contract as production responses.311- Treat sandbox/production parity as a high-priority regression target.312- In sandbox/spike mode, write tests for bugs found — coverage grows organically. For production refactors, see §1.5 (tests required for behavior changes).313314---315## 6.7 Test-Induced Production Defense Detection316317**Rule:** Do not add production defensive code solely to satisfy unrealistic tests. A production guard is allowed only when the invalid state can occur at a real boundary or represents an explicit domain rule.318319| Production change smell | Likely test problem | Required action |320|---|---|---|321| Internal `if (!x) return` added after unit test fails | Test fixture omitted required field | Fix fixture factory or test boundary validation |322| Required field made optional to satisfy test | Test is using invalid domain object | Restore required type and update test data |323| Catch-all added so test passes | Test expects silence instead of failure | Assert typed error or user-visible failure |324| Production default added for impossible state | Test bypassed constructor/parser | Use real constructor/parser in test |325| Private helper exported only for test | Test is coupled to implementation | Test public behavior or move helper to test support |326| Sleep/retry added only for test flake | Test lacks deterministic synchronization | Wait on observable condition or fake clock |327| `NODE_ENV === "test"` branch added | Test-only production behavior | Remove branch; improve test harness |328329**Required questions before adding a guard:**3301. Is the input from an untrusted boundary? → If yes, validate at that boundary3312. Can this state happen in production? → If no, fix the test3323. What contract allows this value? → Cite schema/type/domain rule3334. Would this hide a real bug? → If yes, fail fast instead334335**Banned patterns:** `process.env.NODE_ENV === "test"` branches, silent fallbacks for impossible internal state, making required types optional for mocks, exporting internals only for tests.336337**Allowed guards:** Boundary validation (process/network/user/file boundary), backward compatibility (documented old schema), security checks, domain invariants, observed production bug regressions, external dependency adapters.338339---340## 7. Accessibility Testing341342### Component Level343- jest-axe / vitest-axe: run axe-core on rendered components344 ```ts345 import { axe, toHaveNoViolations } from 'jest-axe'346 expect.extend(toHaveNoViolations)347 expect(await axe(container)).toHaveNoViolations()348 ```349350### Page Level351- Playwright a11y assertions:352 ```ts353 import AxeBuilder from '@axe-core/playwright'354 const results = await new AxeBuilder({ page }).analyze()355 expect(results.violations).toEqual([])356 ```357358### CI Pipeline359- **Gate order (verified 2026-07-02):** component axe (jest-axe/vitest-axe) → page axe360 (@axe-core/playwright) → keyboard/focus/manual checks. **Blocking gate = zero361 serious/critical axe violations + targeted manual checks.**362- Lighthouse a11y score (≥90) is an advisory route-level smoke signal, NOT the blocking363 gate — Lighthouse itself separates automated scoring from required manual checks.364- Pa11y: page-level scanning for WCAG AA violations365- Run a11y tests on EVERY page route, not just the homepage366367### Observability Verification368369Verify trace propagation in integration tests. Assert that spans appear for critical paths. Check structured log format matches the schema in `../jaw-dev-backend/references/core/observability.md`.370371---372373## 8. Security Testing374**→ Delegated**: threat modeling and secure design policy belong to `jaw-dev-security`.375This section covers the **automated test hooks and CI gates** that enforce those rules.376### 8.1 Minimum Security Stack377```text378fast local checks379→ Semgrep / CodeQL gate380→ dependency audit381→ auth / validation regression tests382```383### 8.2 Dependency Scanning Commands384```bash385npm audit --audit-level=high386pip-audit --strict --desc387```388### 8.3 Semgrep Gate389The `returntocorp/semgrep-action` wrapper is deprecated (stated by the repo itself) —390run Semgrep natively in CI:391```yaml392semgrep:393 runs-on: ubuntu-latest394 container: semgrep/semgrep395 steps:396 - uses: actions/checkout@v4397 - run: semgrep ci --config p/default --config p/javascript --config p/typescript --config p/python398```399(Open-source alternative engine: Opengrep, the LGPL-2.1 community fork — see400`../jaw-dev-security/references/static-analysis.md`.)401### 8.4 Security Regressions402Test missing auth (expect 401) and verify error.code matches contract for every auth-protected endpoint.403### 8.5 Rules404- dependency audit in CI405- Semgrep or equivalent SAST406- auth / permission regression tests407- validation tests for malicious or malformed input408- a blocking rule for high / critical dependency findings409---410## 9. Coverage & Quality Gates411### 9.1 Suggested Thresholds412These are project/risk-based, not universal minimums. Adjust for your context.413414| Metric | Suggested Floor | Ideal |415|--------|-----------------|-------|416| Line coverage | 70% | 85%+ |417| Branch coverage | 60% | 80%+ |418| Function coverage | 80% | 90%+ |419| Diff coverage | 80% | 90%+ |420### 9.2 Outcome Metrics421| Metric | Target |422|--------|--------|423| Defect detection rate | > 80% |424| Mean time to detect | < 1 CI run |425| Test signal-to-noise | > 95% |426| Contract drift rate | near 0 |427### 9.3 Coverage Workflow4281. generate coverage reports429 ```bash430 npm test -- --coverage431 npx vitest run --coverage432 pytest --cov --cov-report=xml433 ```4342. review by priority: auth, payment, mutations, upload, contracts first4353. write targeted tests for the gaps4364. publish artifacts and fail the merge when thresholds drop437### 9.4 Quality Gate Checklist438- [ ] focused unit / service tests439- [ ] API integration tests for changed routes440- [ ] contract tests for shared payload changes441- [ ] Playwright smoke for critical rendered journeys442- [ ] security scan / dependency scan443- [ ] coverage thresholds and diff coverage444- [ ] CI artifacts uploaded for failure analysis445446### 9.5 Limited-Oracle / Score-Objective Evaluation447448Use these rules when the true evaluator is scarce, paid, rate-limited, or opaque and449local checks are only proxy metrics for a score/objective. PABCD loop response to450repeated candidate deaths is owned by `jaw-dev-pabcd` §10 Optimization-Loop Meta-Rules.451452- **STRICT (GATE-ORACLE-VALIDITY-01):** When the true evaluator/oracle is rate-limited453 (limited submissions, paid runs) and local metrics are proxies, evaluator validity is a454 prerequisite gate. Before trusting the proxy for accept/reject, quantify historical455 divergence: cases where the proxy said better/equal but the oracle said worse. A proxy456 with known optimistic bias must not be the sole acceptance evidence.457- **DEFAULT (GATE-PREFIX-HORIZON-01):** Replay-based evidence (recorded logs, scripted458 opponents) is prefix-valid only; it stops being valid when the candidate diverges from459 the recorded trajectory. Candidates that diverge early need live adversarial evaluation460 through a modeled opponent/environment, not replays. State the divergence turn/point461 when citing replay evidence.462- **DEFAULT (GATE-INVARIANT-EV-01):** Every hard invariant in an acceptance gate (a metric463 that must not regress) needs an expected-value justification: protected value versus the464 candidate-space it vetoes. If a hard invariant vetoes 3+ consecutive candidates that465 target strictly larger gains, downgrade it to a soft cost and re-justify or remove.466- **DEFAULT (GATE-HOLDOUT-LEAKAGE-01):** Fixed evaluation sets become training data467 under adaptive reuse. Rotate or quarantine repeatedly used sets, keep a blind final468 set, and treat repeated score-driven tuning on the same instances as leakage risk.469 Grounding: Blum and Hardt, "The Ladder: A Reliable Leaderboard for Machine470 Learning Competitions" (arXiv:1502.04585), and Dwork et al., "The reusable holdout:471 Preserving validity in adaptive data analysis" (arXiv:1506.02629).472- **HEURISTIC (GATE-AGREEMENT-STATS-01):** For proxy-vs-oracle validation, report473 agreement statistics that expose error shape: sign-discordance, bias, and worst-case474 error. Correlation alone is not enough; Bland-Altman style agreement analysis is the475 grounding model.476477Grounding: observed in a 14-discard optimization plateau where a prefix-only replay gate478and a hard draw-protection invariant locked a 3.5/8 score.479---480## 10. Pre-Flight Test Checklist481### 10.1 Change-Type Routing482- [ ] pure business logic change → add / update unit or service tests483- [ ] API or middleware change → add / update API integration tests484- [ ] shared frontend↔backend payload change → add / update contract tests485- [ ] rendered user flow change → add / update Playwright smoke coverage486- [ ] auth / upload / billing / external integration change → add security or edge-case regression coverage487### 10.2 Harness Readiness488- [ ] fixtures are deterministic and reusable489- [ ] real dependencies are used where correctness matters490- [ ] Testcontainers are used for DB truth, not mocked SQL491- [ ] external APIs are mocked or recorded intentionally, not accidentally called live492- [ ] `ENFORCE_TDD` requirements were followed if enabled493### 10.3 Contract & Data Integrity494- [ ] response envelope remains stable or contract was updated first495- [ ] error codes are asserted, not only HTTP status496- [ ] `requestId`, pagination, and nullability are verified where relevant497- [ ] frontend fixtures do not drift from backend examples498### 10.4 CI & Reporting499- [ ] relevant CI jobs exist and are actually executed500- [ ] sharding / matrix choices match project size501- [ ] flaky failures were investigated instead of blindly retried502- [ ] coverage / junit / trace artifacts are available on failure503### 10.5 Final Rule (risk-tier)504Verification intensity follows the work class (`jaw-dev` §0.0 / `references/core/crud-test-matrix.md`):505for C2 UI work, one focused smoke (manual click-through or one Playwright run) plus targeted506checks IS a complete story; for C3/C4 or release-sensitive work, a single smoke is not enough —507run the affected suites and required negatives. Manual/Playwright smoke is a risk-tier rule,508not a universal blocker.509```text510unit / service511→ API integration512→ contract verification513→ Playwright smoke514→ CI gate + coverage + security scan515```516## Patch Integrity Gate (TEST-PATCH-INTEGRITY-01, DEFAULT)517518Source: sol research (SWE-bench containerized evaluation, addyosmani/agent-skills).519520An agent that obtains green by weakening tests has not fixed the bug. Before521claiming implementation complete:5225231. **Baseline**: record which tests fail and their failure signatures BEFORE any524 production code change.5252. **Implement**: write the fix/feature.5263. **Re-run originals**: execute the ORIGINAL test suite (not the modified version)527 against the patched source. All baseline failures must now pass.5284. **Classify test changes**: every test/config change is `required` (new test for529 new behavior), `suspicious` (deleted assertion, lowered threshold, added skip,530 reduced coverage exclusion), or `unrelated`.5315. **Justify suspicious changes**: each suspicious change needs a stated reason.532 "The test was wrong" is valid only with evidence of the original test's incorrectness.533534Red flags that trigger escalation:535- Deleted assertions without replacement536- Snapshot updates without visual/behavioral verification537- Coverage exclusions added in the same PR as the fix538- `@skip` or `.skip()` added to failing tests539- Threshold reductions (e.g., coverage 80% → 60%)540- Type assertion suppressions (`as any`, `@ts-ignore`) in test files541542## TDD Evidence Contract (TEST-TDD-EVIDENCE-01, DEFAULT)543544When TDD is claimed, durable evidence must show:545- RED: failing test name + failure message (before production code)546- GREEN: same test passing (after production code)547- REFACTOR: full affected suite passing (after cleanup)548549A TDD claim without RED evidence is not TDD.550---551552## Acceptance-Row Reachability (TEST-ROW-REACHABLE-01, DEFAULT)553554Every row of an acceptance-criteria table must have a **constructible precondition**.555Before writing a row, ask: is there a call path that reaches this state? Does an556earlier guard consume this condition first, so the branch under test is never entered?557Does an operation that produces this value actually exist?558559An unreachable row is decoration, not verification — the implementer tries to write560that test, cannot, and quietly drops it. This applies C-ACTIVATION-GROUNDING-01's561requirement (every conditional path names its activation scenario) to each row of the562acceptance table, not just to the plan's prose.563564Common unreachable shapes:565566- The row asserts a rejection that an earlier, broader rule already rejects — the567 specific guard is never exercised, so the test passes even if that guard is absent.568- The row needs a state the public API cannot produce (no operation creates it).569- The row asserts on a call-site argument the function never receives.570- Two rows in the same table are mutually exclusive on the same tree (for example571 "the gate reports a failure here" together with "the gate exits 0 overall").572573## Test Oracle Integrity (DEFAULT)574575Three narrow contracts that stop false-green. No gate enforces any of them; the576reviewer is the only check.577578**TEST-PROMPT-SEAM-01.** Do not assert on prose. A test may read a document only when579it **extracts a value and compares it against a value from another source.** Asserting580that a phrase exists in a file is a violation no matter how many files you read or581what the test's header says it is for.582583- Forbidden: `assert.match(readFileSync(".../SKILL.md"), /prefer rg first/i)` — one584 source, phrase existence, breaks on harmless rewording, proves no behavior.585- Allowed: parse the frontmatter `description`, pull a token out of it, and assert that586 token appears in the runtime constant it is supposed to mirror — two sources, values587 compared, breaks only when they genuinely disagree.588- Outside this rule's scope, and allowed: asserting on non-prose values — version pins,589 license names, runtime output, CLI stdout, file existence.590591The rule was written after two such tests were **deleted rather than repaired**, and592the reason repair failed is the reusable part. Three designs were tried: a structured593metadata field in each document compared between them (nothing at runtime reads such a594field, so it becomes a third source of truth that passes whenever both copies are wrong595together); promoting the prose to a behavioral test (the behavior was already covered,596so this duplicated coverage while appearing to offset the deletion); and promoting a597doctrine statement the same way (no runtime branch implements it, so there is nothing to598compare against).599600What that costs is stated plainly rather than hidden: the affected doctrine has **no601automated consistency check** and is a human-review item. **When prose has no602counterpart in code, a test that reads it can only check that the words are still603there.** Deleting it removes a false green; inventing a second document to compare it604against removes nothing and adds a lie.605606**TEST-ORACLE-INDEPENDENCE-01.** Never derive the expected value from the code under607test.608609- Forbidden: `assert.equal(fn(x), fn(x))`; building the expectation with a helper the610 code under test also uses; refreshing a snapshot from current output and calling that611 verification.612- Allowed: hardcode the expectation in the fixture, or compute it by an independent613 route — a second implementation, a hand-worked example, an external spec.614615**TEST-PRECEDENCE-FIXTURE-01.** When testing override / default / fallback, the three616values must all **differ** — otherwise the test cannot tell which path ran.617618- Forbidden: override `"x"`, default `"x"`, fallback `"x"` — every branch passes.619- Allowed: override `"from-flag"`, default `"from-config"`, fallback `"builtin"`, with620 each case asserting the specific one.621622A regression test should FAIL when the defect is reintroduced. Confirm it once by623mutation — break it, watch it go red, restore it, watch it go green. A regression test624that has never been seen red is an assertion about the test, not about the defect.625626## CI Green Discipline (TEST-CI-GREEN-01, STRICT)627628The latest HEAD is the source of truth for whether CI is green.6296301. **Inspect before editing.** Read the failing job's log and its artifacts. A failure631 you have not read is a failure you are guessing about.6322. **Make the minimal correct fix** for that failure.6333. **Re-watch the latest HEAD**, not the run you were looking at when you started.634635Never blind-retry a failed job, and never push another change without new failure636evidence. Both produce the same illusion — activity that looks like progress while the637failure is unexamined. Re-running to measure a failure RATE is a different act and is638allowed; see `references/ci-pipeline.md` §5.2.639640## Exploratory QA Tier (TEST-CU-QA-01, DEFAULT)641642Deterministic browser test suites and exploratory QA are different tools for different643jobs, and the split is worth stating because collapsing it wastes both:644645- **Deterministic suites** own regression guarding. They are the thing you keep.646- **Exploratory QA** through whatever browser or screen-control capability the runtime647 offers owns immediate proof that a surface built right now actually works.648649Protocol: **inspect → act → re-inspect.** When DOM inspection is unavailable, fall back650to screenshots **and read the image back** — a screenshot produced but never read is651not an observation.652653Evidence names the flow, the states traversed, the result, and the artifacts. **Promote654any flow that must stay guarded into a deterministic test**; exploratory proof does not655accumulate, so a flow left only in QA is a flow that will regress unnoticed.656657The tool ladder for this tier is `QA-TOOL-LADDER-01` in658`../jaw-dev/references/browse-qa-ladders.md`; use an available faithful659capability and state why a ladder step was skipped.