# Playwright Validator

> Drive a deployed frontend via Playwright using OpenSpec WHEN/THEN scenarios; emit behavioral_failure findings.

- Skill: `jankneumann/playwright-validator` (Agent Skill, multi-file: 10 files)
- Install (CLI): `npx skillmds@latest add jankneumann/playwright-validator`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jankneumann/playwright-validator/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: jankneumann (https://skillmd.com/u/jankneumann)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/jankneumann/playwright-validator

---


# Playwright Validator

Behavioral validator for frontend surfaces. Reads OpenSpec scenarios
(`openspec/changes/<id>/specs/**/*.md`) and a frontend descriptor
(`evaluation/descriptors/<id>.yaml`), generates Playwright
TypeScript test files, executes them with `npx playwright test --reporter=json`,
and emits `findings-playwright.json` conforming to
`openspec/schemas/review-findings.schema.json`.

This is the **peer skill** chosen in design D2 of
`factory-missions-architecture-alignment` — packaged separately from
`packages/gen-eval/` so its Node/browser-binary
dependencies don't bleed into the gen-eval Python runtime.

## When to use

* You have a frontend at a URL (or a static fixture) and an OpenSpec change
  with `#### Scenario:` blocks describing click-and-assert flows.
* You want behavioral evidence (deployed-system behavior) to feed into
  `consensus_synthesizer` alongside scrutiny reviewer findings.
* `validate-feature --phase gen-eval` auto-detects a `frontend-descriptor`
  and dispatches here.

For HTTP/MCP API surfaces, use `/gen-eval` instead — that path remains the
non-frontend dispatch target.

## Arguments

`$ARGUMENTS` — at minimum, the change-id:

* `<change-id>` (required, `^[a-zA-Z0-9_-]+$`) — OpenSpec change identifier.
* `--descriptor PATH` — path to frontend-descriptor YAML (default:
  `evaluation/descriptors/<change-id>.yaml`).
* `--specs-dir PATH` — override OpenSpec specs directory.
* `--output-dir PATH` — where `findings-playwright.json` is written
  (default: `openspec/changes/<change-id>/`).
* `--test-dir PATH` — where the generated `.spec.ts` files land
  (default: `skills/playwright-validator/.generated/<change-id>/`). Must not
  be a directory named `test-results` — that is Playwright's own `outputDir`,
  which it clears before each run, deleting the generated specs.
* `--browsers chromium [firefox webkit]` — override descriptor's matrix.
* `--dry-run` — emit the `.spec.ts` but do not invoke `npx playwright test`.

## Invocation

Direct (Python):

```bash
python3 "<skill-base-dir>/scripts/cli.py" <change-id>
```

Module form (after the skill is on `PYTHONPATH`):

```bash
python -m playwright_validator <change-id> [--descriptor PATH] [--output-dir PATH]
```

Dispatch shim (used by `validate-feature --phase gen-eval`):

```bash
bash "<skill-base-dir>/scripts/dispatch.sh" <change-id>
```

## Exit codes

| Code | Meaning |
|------|---------|
| 0    | All scenarios passed |
| 1    | One or more Playwright tests failed; `findings-playwright.json` emitted |
| 2    | Pipeline error: missing descriptor, malformed YAML, missing env var |
| 64   | Invalid change-id (failed `^[a-zA-Z0-9_-]+$`) |
| 127  | Playwright CLI not on PATH; **no findings file** emitted |

## Auto-detection in validate-feature

`validate-feature --phase gen-eval` walks
`evaluation/descriptors/*.yaml` and routes each descriptor:

1. If the YAML validates against
   `contracts/frontend-descriptor.schema.json` → dispatch to this skill via
   `dispatch.sh`.
2. Otherwise → existing HTTP/MCP gen-eval path
   (`packages/gen-eval/`).

The detection predicate is `descriptor.is_frontend_descriptor(path)` from
`<skill-base-dir>/scripts/descriptor.py`.

## Findings output

Per `contracts/findings-vendor-source.md` and design D8, this skill
**always** writes to `findings-playwright.json` (never `findings-gen-eval.json`).
The two files coexist when both validators run on the same change.

Each finding:

* `type: "behavioral_failure"` (per design D3).
* `criticality: "high"` (default for failed Playwright assertions).
* `file_path` + `line_range` reference the **OpenSpec spec.md**, not the
  generated `.spec.ts` file (per the "Playwright findings trace to OpenSpec
  scenarios" spec scenario).
* `metadata.browser` — chromium / firefox / webkit (per the "Browser matrix"
  scenario).
* `metadata.scenario_id` — the OpenSpec scenario's name.

## Sample frontend (smoke test)

A minimal static HTML page is available at
`packages/gen-eval/tests/fixtures/sample-descriptor.yaml` (per design D7 —
package-shipped data). To exercise the full pipeline (requires Node + Playwright):

```bash
python3 "<skill-base-dir>/scripts/cli.py" \
    sample-frontend-demo \
    --descriptor packages/gen-eval/tests/fixtures/sample-descriptor.yaml \
    --specs-dir openspec/changes/sample-frontend-demo/specs
```

## Localhost-bind invariant (D7)

The descriptor schema's `lifecycle.bind_address` defaults to `127.0.0.1`.
The runner additionally inspects `lifecycle.startup_command` and refuses to
launch when `0.0.0.0` appears unless the operator explicitly set
`bind_address` to a non-localhost value.

## Auth-flow env vars

`auth_flow[].value` may reference `${VAR_NAME}` (regex `^[A-Z_][A-Z0-9_]*$`).
Substitution uses Python's `string.Template` — never shell expansion. A
missing env var causes the pipeline to abort **before any browser launches**
with the exact error:

```
auth_flow: required env var <VAR_NAME> not set
```

Operators can preflight by setting the optional `env_vars_required` list in
the descriptor; the validator checks it before the auth_flow walk.

## Testing

Unit + integration tests at `skills/tests/playwright-validator/`. Run:

```bash
# Source-contribution-only validation from the agentic-coding-tools checkout
# (not an installed runtime command):
python3 -m pytest skills/tests/playwright-validator/ -v
```

Tests that would invoke real `npx playwright` skip gracefully when the CLI
is unavailable (`pytest.skip("requires npx playwright")`).