Initialize Test Suite Project

Initialize Test Suite Project

Scaffold or migrate a test-suite project — API, E2E/UI, or shared test library. Owns test-suite-specific archetype selection, shared-library vs in-repo scaffolding decision, layer variants, tag taxonomy, reporter stack, and custom report. Cross-cutting concerns (adoption level, personas, AGENTS.md base template, i18n, knowledge graph, roadmap nudge, final commit) delegate to harness-initialize-project.

When to Use

• Initializing a new test-suite project (you already know it is a test suite)
• Migrating an existing test-suite project to the next adoption level
• When harness-initialize-project's Phase 1 classification step hands off here
• NOT when the project is a product or service (use harness-initialize-project)
• NOT when adding a single spec, domain, or fixture to an already-configured test suite (use add-harness-component)

Composition with harness-initialize-project

This skill owns only the test-suite-specific pieces. The generic harness flow belongs to the parent skill. Two invocation patterns:

1. Parent dispatches here. The user runs harness-initialize-project; its Phase 1 step 5 classifies the project as a test suite and dispatches to this skill. Run the parent's Phase 2 (scaffolding) first, then this skill's full flow, then return to the parent for Phase 4 wrap-up (knowledge graph, roadmap, commit).

2. Direct invocation. The user knows up front it's a test suite and invokes this skill directly. Run the parent's Phase 1 (assess state), Phase 2 (harness init), and Phase 3 steps 1–2 and 5 (personas, AGENTS.md base, i18n) inline or by invoking the parent explicitly, then apply this skill's Phase 1–4, then hand back to the parent's Phase 4 steps 4–5.

Either way, what this skill owns: the canary probe (Phase 0) and canary-vs-built-in delegation, archetype selection, shared-library vs scaffold decision, layer-model variants A/B, ESLint flat-config fix, test-framework delegation, env and secrets, auth abstraction, schema validation, fixtures isolation, mocking support, tag-driven organization, reporters, the custom report, CI integration, and "prove the guards fire" verification.

Process

Phase 0: PROBE — Detect canary CLI availability

The deterministic canary test CLI (canary-test-cli) now owns two things this skill historically hand-rolled: framework selection (classifier → recommender) and the test-reporter contract that CI-readiness tooling reads. Probe for it once, up front, so the rest of the flow can delegate when canary is present and degrade to the built-in flow when it is absent (this skill ships to adopter projects that may not have canary).

Call the canary_probe MCP tool once. It returns { status: "available" | "degraded", version?, reason? } and never errors.

• available — the deterministic canary CLI is usable. Delegate for the rest of the run:

  • Framework selection → canary_recommend_framework + canary init (Phase 3 step 4). Do not make a manual human framework choice.
  • Reporter → the canary test-reporter version:1 contract via canary init (Phase 3 step 11), so the new suite is born legible to canary's ci-ready check. Do not scaffold the bespoke scripts/generate-reports.ts.

• degraded — the CLI is not usable (reason: not-installed, binary-missing, exec-failed, or bad-output). Print one line:

  canary CLI unavailable (<reason>) — install canary-test-cli for deterministic framework recommendations and the shared reporter contract. Proceeding with the built-in flow.

  Then use the built-in flow for the rest of the run: framework delegation to test-playwright-setup / test-vitest-config (Phase 3 step 4) and the bespoke scripts/generate-reports.ts scaffold (Phase 3 step 11).

Record the probe result in AGENTS.md so later agents know whether the suite is canary-backed.

Phase 1: CLASSIFY — Pick the Archetype

Before scaffolding, classify the project and record in the assessment.

Test-suite signals (at least one must apply — re-check even if dispatched from the parent):

• Repo or package name matches *test*, *-e2e*, *-qa*, *-automation*
• package.json has @playwright/test, cypress, webdriverio, mocha, or testcafe as a direct dep (frameworks that are only test-related — presence of vitest/jest alone is not enough)
• Top-level tests/, e2e/, specs/, or playwright/ directories are the primary source tree, not an adjunct to src/
• Config files like playwright.config.*, cypress.config.*, wdio.conf.*
• No production runtime — build output is consumed only by other test repos (shared library)

Archetypes:

• API test suite — exercises HTTP APIs end-to-end (e.g. Playwright API tests)
• E2E / UI suite — browser-driven tests (Playwright, Cypress, WebdriverIO)
• Shared test library — API clients, Page Object Models, fixtures, or test users consumed by the above; no tests in-repo

Phase 2: DECIDE — Shared Library or Scaffold In-Repo?

For API and E2E/UI suites, check whether a shared library already exists — typically a team-specific @<org>/<team>-testing-api-library for API clients, or an equivalent for Page Object Models. If one exists, consuming it is strongly preferred: one client implementation shared by all test suites is the contract source of truth and avoids drift. Only scaffold an in-repo src/api/ or src/page-objects/ tree if no shared library exists or the suite has a genuinely different contract.

For Shared test library archetypes, this phase is trivial — you are the shared library; scaffold in-repo.

Record the decision in AGENTS.md. It determines which layer-model variant applies in Phase 3 step 2.

Phase 3: CONFIGURE — Test-Suite Shape

Apply these steps before running validation. Record decisions in AGENTS.md as you go.

1. Pick a domain layout. Independent of whether clients are in-repo or imported, domains drive the folder layout:

   • Self-contained suite: one folder per API domain, feature area, or user flow under src/api/ or src/page-objects/, each exposing a manager/client class (API suites) or Page Object Model (UI suites) that takes an HTTP client or browser context in its constructor. A single composition module wires everything together.
   • Shared-library consumer: no per-domain folders in src/. Instead, a thin config/ adapter wraps the shared library's entry point (e.g. exposes {api, user} on the Playwright test fixture). Tests are still organized by domain — see step 10 — but the domain folders live under tests/<domain>/, not src/api/<domain>/.

   Document the chosen layout in AGENTS.md with a directory tree.

2. Define the layer model. Two variants — pick the one matching Phase 2.

   Variant A — self-contained test suite. Record in harness.config.json:

   • utils → helpers, date/string/id utilities (no internal deps)
   • composition → single wiring module (e.g. managers.ts) → allowed to import every layer below
   • common → shared DTOs, enums, error types → utils
   • config → HTTP client, auth service, environment loader → utils, common
   • api / domains / page-objects → per-domain managers or POMs → utils, common, config
   • fixtures → deterministic seed data, factories, builders → utils, common, config
   • specs → top layer; imports composition + fixtures; nothing imports specs

   Forbid (same-layer rules, enforced via forbiddenImports):

   • A domain/POM importing a sibling domain/POM (domains are peers, not hierarchical)
   • fixtures importing specs
   • Specs importing each other (tests must be independently runnable in any order)

   Variant B — shared-library consumer. No common, no api layer — those live in the shared library. Record in harness.config.json:

   • utils → helpers with no internal deps
   • config → thin adapter wrapping the shared library (e.g. builds the Managers from the shared lib + current env), token handler, constants → utils
   • fixtures → Playwright test fixtures that expose {api, user, ...} to specs → utils, config
   • specs → top layer; imports fixtures → utils, config, fixtures

   Forbid:

   • Specs bypassing fixtures and instantiating the shared library directly — always go through the fixture so auth, base URL, and teardown are consistent
   • fixtures importing specs
   • Any layer importing the shared library from outside config (forces single adapter point, keeps upgrades localized)

   Layer-ordering gotcha — harness check-deps and @harness-engineering/eslint-plugin both use first-match-wins layer resolution. Put more-specific patterns before more-general ones in the layers array. For Variant A, composition (pattern src/config/managers.ts) must come before config (pattern src/config/**), otherwise managers.ts is classified as config and every api import becomes a layer violation. Do not rely on per-layer excludePatterns — the CLI and the ESLint plugin both drop that field.

   forbiddenImports gotcha — no exceptPaths support. The ESLint plugin schema for forbiddenImports only accepts from, disallow, message — any other keys (including exceptPaths) are silently stripped. Do not write from: src/config/**, disallow: [src/api/**], exceptPaths: [src/config/managers.ts] and expect it to work. Use layer ordering (above) to carve out the composition file; reserve forbiddenImports for same-layer rules like sibling-domain isolation that the layer graph cannot express.

3. Wire up the ESLint flat config so the forbidden-imports rule actually runs. harness init scaffolds eslint.config.mjs that spreads harnessPlugin.configs.recommended but does not declare a files: glob or a TypeScript parser. Under ESLint 9 flat config this means every .ts file is ignored and the rule never fires. Replace the scaffolded config with an explicit block: files: ['src/**/*.ts', 'tests/**/*.ts'], languageOptions: { parser: tsParser, parserOptions: { ecmaVersion: 'latest', sourceType: 'module' } }, plugins: { '@harness-engineering': harnessPlugin }, and the three 'error' rules (no-layer-violation, no-forbidden-imports, no-circular-deps). Add @typescript-eslint/parser to devDependencies. Verify by running npx eslint 'src/**/*.ts' and seeing it actually report errors on a deliberately bad file.

4. Pick the test framework(s). Which path you take depends on what Phase 0 reported.

   When Phase 0 reported available (canary present): do not pick the framework by a manual human choice — canary owns this via its classifier → recommender. Call the canary_recommend_framework MCP tool with a prompt describing the suite (archetype + what it exercises, e.g. "end-to-end browser login flow" or "HTTP API contract tests for the rewards service"); it returns { framework, test_type, file_extension, reasoning[], alternatives[] } deterministically (no API key, no LLM round-trip). Record the recommended framework and its reasoning in AGENTS.md, then scaffold it with canary init so the suite is born with canary's project shape and reporter wiring. Drop into the follow-up skills below only for framework-specific pattern depth that canary does not scaffold (e.g. test-playwright-patterns, test-msw-pattern).

   When Phase 0 reported degraded (canary absent): fall back to today's flow — do not configure the framework here, just record the choice in AGENTS.md and point the human at the follow-up skill:

   • Playwright (API or E2E) → test-playwright-setup then test-playwright-patterns
   • Vitest → test-vitest-config
   • Contract tests → api-contract-testing and test-contract-testing
   • Mocking (MSW, route interception) → test-msw-pattern, test-mock-patterns
   • Fixtures / factories → test-factory-patterns, harness-test-data

   Either way, for shared libraries that are consumed by tests but contain none themselves, skip framework setup entirely — record this in AGENTS.md under "Gotchas" so agents do not try to add Vitest later.

5. Environment and secret handling. Require a committed .env.example listing every variable the suite reads (API base URLs, OAuth client IDs, registry tokens, feature flags). Actual .env* files must be gitignored. For multi-environment suites, encode environments as .env.dev, .env.stage, .env.prod and document in AGENTS.md how the runner selects one (env var, CLI flag, or config). Never commit real credentials even as examples — use placeholders like REPLACE_ME.

6. Authentication abstraction. Record in AGENTS.md: token source (OAuth client-credentials, CIAM, basic auth, session cookie), caching strategy (in-memory, file, Redis), and how test users or parent/child scopes are resolved. If a shared test-user library exists (typically @<org>/<team>-testing-user-library or similar), link it and forbid inlining user credentials in specs.

7. Schema / contract validation. Decide whether request and response DTOs are validated at runtime (zod, ajv, io-ts) or only compile-time. Intermediate and above should validate at runtime for contract-drift detection. If adopting zod, reference zod-schema-definition.

8. Fixtures and test-data isolation. Document the per-test isolation strategy in AGENTS.md: unique IDs per run (UUID suffixes, timestamps), teardown hooks, no shared mutable state between specs. Forbid hard-coded production record IDs in fixtures.

9. Mocking support (shared libraries and API clients). If the project will be consumed by FE or UI test suites that need to mock upstream responses, expose mock response builders (e.g. buildMockResponse, buildMockErrorResponse) and document the override API in AGENTS.md so consumer projects know the contract.

10. Test organization: tags over folders. Do not split specs into tests/smoke/, tests/regression/, etc. Organize specs by domain / feature area (tests/<domain>/<feature>.spec.ts) and filter run shape with tags. This matches how Playwright (and most modern runners) expect grep-based filtering to work, and avoids moving a spec file whenever its coverage tier changes.

    Record the tag taxonomy in AGENTS.md. Baseline for a Playwright API suite:

    • @smoke — fast subset; PR gate. Applied per-test or per-describe.
    • Untagged = regression. "Full suite" is just playwright test with no grep.
    • Quarantine tags (never run): @known-failure, @wip, @blocked, @archived. Exclude these globally via grepInvert in playwright.config.ts:

      grepInvert: [/@known-failure/, /@wip/, /@blocked/, /@archived/],
      

    Filter commands (bake into package.json scripts, not ad-hoc CI yaml):

    • npm run smoke → playwright test --grep @smoke
    • npm run exhaustive → playwright test (full regression, quarantine excluded)
    • npm run list → playwright test --list (sanity check after tag edits)

    Reporters. Configure playwright.config.ts with the stack that feeds both human and machine consumers:

    reporter: [
        ['html', {outputFolder: './test-results/html-test-results', open: 'never'}],
        ['list'],
        ['github'],
        ['json', {outputFile: 'test-results/json-results/results.json'}],
    ],
    

    The json reporter is the input to the custom report in step 11. The github reporter produces annotations on PRs; list is for local stdout; html is for deep-dive failure triage.

11. Custom report (enriched HTML + Slack + history). Playwright's stock HTML report is fine for a single run but does not track trends, categorize failures, or know which tests are quarantined.

    When Phase 0 reported available (canary present): do not scaffold a bespoke reporter — canary owns this. canary init wires the canary test-reporter version:1 contract, which already emits exactly the enriched output below (failure categorization, per-area health, delta-vs-history JSONL, and the quarantine ledger) in the shape canary's ci-ready check reads. Keeping the reporter on canary's contract means the new suite is born legible to canary's CI-readiness tooling instead of drifting from a hand-rolled schema. Record in AGENTS.md that reporting is canary-owned (test-reporter version:1) and that CI-readiness is checked with canary's ci-ready; do not add scripts/generate-reports.ts.

    When Phase 0 reported degraded (canary absent): scaffold a scripts/generate-reports.ts that reads the JSON reporter output (test-results/json-results/results.json) and emits an enriched report. Modeled on reference implementations running in production test suites, it should produce:

    • Failure categorization: bucket each failure into one of schema | auth | server | client | timeout | network | other based on the error message. Surfaces whether a red run is an API regression (schema) vs. infrastructure (timeout/network) vs. test-data problem (auth).
    • Per-area health: group by domain folder (first path segment under tests/), report pass rate and failing titles per area. Tells you which surface of the product is drifting.
    • Delta vs history: append each run to a JSONL ledger keyed by commit. Report new failures since last run and failures since last green — not just "X failed" but "X started failing at commit Y."
    • Quarantine ledger: scan for @known-failure-tagged tests, track age since first seen in a JSON ledger, flag tests quarantined longer than 14 days.
    • Slack summary: short markdown block with pass rate, new failures, and a link to the HTML report. Optional but cheap once the categorization exists.
    • Environment + git context: node version, base URL, commit SHA, branch, commit message — so a report opened later tells you where and when it ran.

    Wire the script into package.json:

    {
      "scripts": {
        "generate-custom-reports": "npx ts-node scripts/generate-reports.ts",
        "show-custom-report": "open test-results/reports/test-report.html"
      }
    }
    

    Do not block CI on the custom report — run it as a post-step and upload the output as an artifact. If categorization throws on an unexpected error shape, the exit code should be non-fatal.

12. CI integration. For intermediate+: PR pipeline runs npm run smoke (tag-filtered, not folder-filtered); main merges trigger npm run exhaustive; nightly runs npm run exhaustive plus a quarantine audit (step 11) reporting quarantined-too-long tests. Record the flake policy (max retries, quarantine threshold) in harness.config.json so agents do not silently raise retries to hide flakes.

Phase 4: VALIDATE — Prove the Guards Fire

Run the three gates on the clean tree:

harness validate
harness check-deps
npx eslint 'src/**/*.ts' 'tests/**/*.ts'

All three must pass before proceeding.

Clean-tree passing is necessary but not sufficient. All three tools can pass while the rules are silently misconfigured (layer ordering wrong, ESLint files glob missing, forbidden-imports schema fields dropped). Before handing back to harness-initialize-project for wrap-up:

• Insert one deliberate forbidden import — a sibling-domain import is the easiest (e.g. src/api/users/users-manager.ts importing ../auth/auth-manager). Run npx eslint <file> and confirm it reports the violation.
• Insert one deliberate cross-layer import (e.g. src/utils/foo.ts importing from src/api/users/). Run harness check-deps and confirm it reports the violation.
• Revert both edits. Confirm the clean tree is green on all three gates.

After verification passes, return to harness-initialize-project's Phase 4 for:

• harness scan (knowledge graph)
• Roadmap nudge
• Final commit

Harness Integration

• harness-initialize-project — parent skill. Owns adoption-level selection, harness init invocation, persona configuration, AGENTS.md base template, i18n decision, knowledge-graph build, roadmap nudge, final commit.
• canary_probe / canary_recommend_framework (MCP) — Phase 0 probes the optional deterministic canary test CLI (canary-test-cli). When available, framework selection delegates to canary_recommend_framework (Phase 3 step 4) and scaffolding + reporter to canary init (test-reporter version:1, read by canary's ci-ready check, Phase 3 step 11). Degrades to the built-in flow (follow-up test skills + bespoke scripts/generate-reports.ts) when canary is absent.
• harness validate — verifies harness.config.json schema, AGENTS.md, layers, persona config.
• harness check-deps — verifies layer-graph dependency constraints at the file level.
• @harness-engineering/eslint-plugin — enforces forbiddenImports (same-layer rules) and surfaces layer-graph violations at lint time.
• Follow-up test skills — test-playwright-setup, test-playwright-patterns, test-vitest-config, test-msw-pattern, test-mock-patterns, test-factory-patterns, harness-test-data, api-contract-testing, test-contract-testing.
• Related libraries — zod-schema-definition (runtime DTO validation).

Success Criteria

• Archetype is recorded in AGENTS.md (API / E2E-UI / Shared library)
• Shared-library decision is recorded in AGENTS.md
• Layer model matches the chosen variant (A self-contained or B shared-library consumer) from Phase 3 step 2
• Layer ordering puts composition before config (Variant A only)
• forbiddenImports uses only {from, disallow, message} — no silently-dropped exceptPaths
• eslint.config.mjs declares an explicit files: glob and imports a TypeScript parser
• .env.example is present with REPLACE_ME placeholders
• No real secrets are committed (real .env* gitignored)
• Test framework choice and tag taxonomy (including @smoke + quarantine tags) are recorded in AGENTS.md
• Specs are organized tests/<domain>/<feature>.spec.ts — no tests/smoke/ or tests/regression/ folders
• playwright.config.ts declares html + list + github + json reporters and grepInvert for quarantine tags
• Phase 0 probed canary once (canary_probe) and recorded the result (canary-backed or built-in) in AGENTS.md
• Framework choice: when canary is available it comes from canary_recommend_framework + canary init; when degraded it is delegated to the follow-up test skill — either way recorded in AGENTS.md
• Reporting: when canary is available it uses the canary test-reporter version:1 contract via canary init (no bespoke scripts/generate-reports.ts); when degraded, scripts/generate-reports.ts exists and is wired to npm run generate-custom-reports
• All three Phase 4 gates pass on the clean tree
• "Prove the guards fire" step completed — deliberate violations were caught and reverted
• Control returned to harness-initialize-project for knowledge-graph build and final commit

Rationalizations to Reject

Examples

Example: Initializing a Shared API Test Library (Intermediate)

CLASSIFY:

Human: "New repo. It's a shared TypeScript HTTP client for our API test suites — consumed by Playwright API tests and also used to mock responses in FE tests. No tests live in this repo."
package.json has no Playwright/Vitest as direct deps (shared library, not a test runner).
Archetype: Shared test library.

PROBE (Phase 0):

canary_probe → { status: "degraded", reason: "not-installed" }
No canary on PATH — proceed with the built-in flow. (A shared library ships no tests,
so framework + reporter are N/A regardless.) Result recorded in AGENTS.md.

DECIDE (Phase 2):

This is the shared library itself — scaffold in-repo. Phase 2 is trivial.

SCAFFOLD (delegated to harness-initialize-project):

harness init --level intermediate --language typescript

CONFIGURE (Phase 3):

Domain layout: one folder per API domain under src/api/ (challenges, rewards,
  user, oauth, ...). Each domain exposes <domain>-manager.ts taking HttpClient
  in its constructor. Single composition module src/config/managers.ts wires
  every domain from one HttpClient.

Layer model (Variant A, harness.config.json):
  utils       → src/utils/**                         (no internal deps)
  composition → src/config/managers.ts               → utils, common, config, api
  common      → src/api/common/**                    → utils
  config      → src/config/**                        → utils, common
  api         → src/api/**                           → utils, common, config

Layer ordering: composition BEFORE config (first-match-wins).

Forbidden (forbiddenImports):
  - any src/api/<domain> importing a sibling src/api/<domain>
  - src/api/common importing config or sibling domains

Framework: none in-repo (shared library). Gotcha recorded in AGENTS.md so
  future agents don't scaffold Vitest.

Env: .env.example committed with GITHUB_ACCESS_TOKEN placeholder (registry
  auth only). No runtime env vars — consumers pass config explicitly.

Auth: OAuth client-credentials with pluggable TokenStorage interface; parent
  vs child scope resolution documented in docs/auth-flow.md. Link to
  a shared test-user library for test-user data.

Schema validation: zod at runtime for all response DTOs. Reference
  zod-schema-definition skill for future domain additions.

Mocking: expose buildMockResponse / buildMockSuccessResponse /
  buildMockErrorResponse from src/config/mock-response.ts so FE test repos
  can override per spec. Document override contract in docs/mock-response.md.

VALIDATE (Phase 4):

harness validate     # Pass
harness check-deps   # Pass — no sibling-domain violations
npx eslint 'src/**/*.ts'  # Pass

# Prove the guards fire: insert a sibling-domain import, eslint reports it, revert.

# Hand back to harness-initialize-project:
harness scan         # Builds .harness/graph/
git commit -m "feat: initialize harness project at intermediate level for shared API test library"

Example: Playwright API Test Suite Consuming a Shared Library (Intermediate)

CLASSIFY:

Human: "New repo. Playwright API tests for one of our backend services. A shared API client library already exists at @<org>/<team>-testing-api-library and a shared test-user library at @<org>/<team>-testing-user-library."
package.json will have @playwright/test as a direct dep → test suite signal.
Archetype: API test suite.

PROBE (Phase 0):

canary_probe → { status: "degraded", reason: "not-installed" }
No canary here — built-in flow: framework delegates to test-playwright-setup,
reporter is the bespoke scripts/generate-reports.ts (shown below). Recorded in AGENTS.md.
(For the canary-present variant of this suite, see the next example.)

DECIDE (Phase 2):

Shared library exists → consume it. Layer variant B (shared-library consumer).
Do NOT scaffold src/api/. Record decision in AGENTS.md.

SCAFFOLD (delegated to harness-initialize-project):

harness init --level intermediate --language typescript
# Scaffolded output is a starting point — replace generic layers and eslint config.

CONFIGURE (Phase 3, Variant B):

Dependencies: @<org>/<team>-testing-api-library, @<org>/<team>-testing-user-library,
  @playwright/test, dotenv, zod.

Directory layout (no src/api/, no src/page-objects/):
  config/
    api-adapter.ts       # Builds Managers from the shared library + env vars
    api-client.ts        # Fetcher wrapping Playwright's APIRequestContext
    token-handler.ts     # OAuth + test-user token resolution
    constants.ts         # TEST_ARTIFACTS_DIR, URLs, etc.
  fixtures/
    test-fixture.ts              # Playwright fixture exposing {api, user}
    test-no-init-auth-fixture.ts # Fixture for 401/auth-failure tests
    expects.ts                   # expectOkResponse, expectUnauthorized, ...
  tests/
    challenges/
      create.spec.ts
      checkin.spec.ts
      ...
    oauth/
      login.spec.ts
    user-profile/
      ...
    global.setup.ts
    global.teardown.ts
  scripts/
    generate-reports.ts
  playwright.config.ts
  .env.example
  harness.config.json
  AGENTS.md

Layer model (Variant B, harness.config.json):
  utils       → src/utils/**        (no internal deps)
  config      → config/**           → utils
  fixtures    → fixtures/**         → utils, config
  specs       → tests/**            → utils, config, fixtures

Forbidden (same-layer or cross-cutting):
  - tests/** importing @<org>/<team>-testing-api-library directly
    (must go through fixtures → config adapter)
  - fixtures/** importing tests/**
  - Specs importing each other

Tags (playwright.config.ts):
  grepInvert: [/@known-failure/, /@wip/, /@blocked/, /@archived/]
  Tag @smoke on PR-gate specs; untagged = regression.

Reporters (playwright.config.ts):
  ['html', {outputFolder: './test-results/html-test-results', open: 'never'}],
  ['list'],
  ['github'],
  ['json', {outputFile: 'test-results/json-results/results.json'}],

Custom report (scripts/generate-reports.ts):
  Reads test-results/json-results/results.json. Emits HTML + Slack with:
  failure categorization, per-area health, delta vs history, quarantine ledger.
  Wired to `npm run generate-custom-reports` / `npm run show-custom-report`.

package.json scripts:
  smoke:                     playwright test --grep @smoke
  exhaustive:                playwright test
  list:                      playwright test --list
  generate-custom-reports:   npx ts-node scripts/generate-reports.ts
  show-custom-report:        open test-results/reports/test-report.html

.env.example: API_BASE_URL, OAUTH_TOKEN_URL, OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET,
  TEST_USER_POOL_URL, TEST_USER_POOL_TOKEN. Real .env is gitignored.

VALIDATE (Phase 4):

harness validate     # Pass
harness check-deps   # Pass
npm run lint         # Pass
npm run smoke        # Sanity: runs only @smoke-tagged specs

# Prove the forbidden-imports rule fires:
# temporarily add an import of @<org>/<team>-testing-api-library to a spec
npx eslint tests/challenges/create.spec.ts  # Should report violation
# revert, confirm green

# Hand back to harness-initialize-project:
git commit -m "feat: initialize harness project at intermediate level for Playwright API suite"

Example: E2E Suite With canary Present (framework + reporter delegated)

Same Playwright suite as above, but canary-test-cli is installed — the Phase 0 probe changes the framework-pick and reporter steps.

PROBE (Phase 0):

canary_probe → { status: "available", version: "1.4.0" }
Canary present → delegate framework selection and the reporter contract to canary.
Recorded in AGENTS.md: "suite is canary-backed".

CONFIGURE (Phase 3, canary-present deltas):

Framework (step 4): do NOT pick manually.
  canary_recommend_framework({ prompt: "end-to-end browser login and checkout flow" })
    → { framework: "playwright", test_type: "e2e", file_extension: ".spec.ts",
        reasoning: ["browser-driven UI flow", ...], alternatives: ["cypress"] }
  Record framework + reasoning in AGENTS.md, then `canary init` to scaffold the
  project shape and reporter wiring. Use test-playwright-patterns only for pattern
  depth canary does not scaffold.

Reporter (step 11): do NOT scaffold scripts/generate-reports.ts.
  `canary init` wires the canary test-reporter `version:1` contract — failure
  categorization, per-area health, delta-vs-history JSONL, and the quarantine ledger
  in the shape canary's `ci-ready` check reads. AGENTS.md records reporting as
  canary-owned; CI-readiness is checked with `canary ci-ready`.

Everything else (layer model Variant B, tags, env, auth) is unchanged from the
canary-absent example above.

VALIDATE (Phase 4):

harness validate     # Pass
harness check-deps   # Pass
canary ci-ready      # Reads the version:1 reporter ledger — suite is legible to CI tooling