# Automated QA Test

> Strict automated QA/backtest workflow for web/app features from a user-written requirement, GitHub issue/PR, bug report, or acceptance criteria. Use when Codex needs to dynamically derive a test plan, run UI/API/WebSocket/SSE/command probes, verify feature logic, interaction behavior, stream completion, persistence/data flow, console/network/runtime errors, screenshots, no-fabrication evidence integrity, completeness checks, and produce an evidence-backed test report. Triggers include 自动化测试, 自动化qa-test, QA test, 回测, 测试需求, issue 测试, PR 测试, 功能逻辑测试, 交互测试, 数据是否通, 接口是否通, WebSocket 流式, SSE 流式, 持久化验证, 错误检查, 严格测试, 不能有遗漏, 禁止编造数据, Playwright 测试报告.

- Skill: `hoshino-wind/automated-qa-test` (Agent Skill, multi-file: 173 files)
- Install (CLI): `npx skillmds@latest add hoshino-wind/automated-qa-test`
- Raw SKILL.md: https://api.skillmd.com/api/skills/hoshino-wind/automated-qa-test/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: Hoshino-wind (https://skillmd.com/u/hoshino-wind)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/hoshino-wind/automated-qa-test

---


# 自动化 QA-Test

Use this skill as a dynamic QA harness. Do not hardcode product pages, route lists, business rules, or expected results inside the skill. Derive them each time from the current request, issue, PR, requirement text, repo behavior, and visible runtime evidence.

## Core Principle

Treat every run as a fresh requirement audit:

1. Read the requirement source.
2. Extract expected behavior, actors, state changes, data dependencies, permissions, edge cases, and success/failure states.
3. Turn those into a test charter and executable test plan.
4. Test feature logic, interaction quality, data/API continuity, and runtime errors.
5. Report what was covered, what passed, what failed, and what remains unverified.

## Agent Loop

Use this skill as an evidence-producing agent loop, not as a static checklist:

1. Compile the requirement into `business-model.json`, `oracle-model.json`, `test-charter.md`, `test-matrix.json`, `test-plan.json`, `qa-metrics.json`, and `closeout-candidates.json`. For a new free-form requirement, start with `scripts/init_qa_artifact.py` or `scripts/scaffold_requirement.py` so the first draft already maps business actors/entities/workflows to UI/API/stream/persistence probes or explicit blockers. Code-review PR text with source/config/test file paths belongs in `code_pr` command-check mode unless the requirement also names explicit runtime UI/API acceptance criteria.
2. Select probes for each matrix row: UI, API, WebSocket/SSE, command/log/persistence, or manual blocker. When `adapter-context.json` exists, run `scripts/synthesize_adapter_probes.py` or `run_qa_cycle.py --synthesize-adapter-probes` before validation to convert known safe adapter paths into executable probes.
3. Run `scripts/preflight_runtime.py` or `run_qa_cycle.py --preflight-runtime` when local services, ports, or project tooling affect execution. Treat preflight blockers as run blockers, not test failures. If the target is a local/test checkout and the user wants the agent to bring services up, run `scripts/service_runtime.py --start` or `run_qa_cycle.py --start-missing-services` so startup PID/log/readiness evidence is captured in `service-runtime.json`.
4. Run `scripts/audit_requirement_coverage.py` or the default `run_qa_cycle.py` coverage gate when `requirement.md` exists. Every requirement-source behavior point must map to `test-matrix.json` before probes run.
5. Run `scripts/validate_plan.py` to catch unmapped matrix requirements, weak probe steps, TODO placeholders, secret-like values, and risky commands before execution.
6. Run `scripts/run_qa_cycle.py` for the full executable loop when using the bundled probe runner.
7. Apply safe concrete follow-up recommendations with `scripts/apply_next_probes.py` or `run_qa_cycle.py --apply-next-probes` before the next execution cycle.
8. Manually update `evidence-ledger.json` only when custom probes outside the runner were used.
9. Generate `qa-verdict.json` with `scripts/generate_verdict.py` or through `run_qa_cycle.py`. Treat `can_claim_pass=false` as the guardrail against calling a failed, blocked, untested, inconclusive, or unaudited run a pass.
10. Generate or write the report only from the audited ledger, verdict, and explicitly labeled gaps.

If the feature spans multiple services or has repo-specific data ownership rules, read `references/project-adapters.md` before planning probes.

When changing or interpreting `business-model.json`, `oracle-model.json`, `qa-metrics.json`, or `closeout-candidates.json`, read `references/business-oracle-model.md`.

## Proof-Carrying Control Boundary

- Treat `qa-run-summary.json`, reports, model text, semantic models, and memory as projections or context, never as authority for `PASS`.
- A normal cycle must hold the run lease, execute through the shared `RunBudget`, commit current outputs to an immutable attempt, append the deterministic terminal state, and pass `scripts/verify_run_proof.py`.
- The outer `qa_agent_loop.py` owns one budget and lease across all iterations. Probe/output consumption never resets between iterations, and a child cycle may inherit only the exact live parent lease.
- Once a managed service is started, ordinary work must preserve the cycle's cleanup deadline/output reserve. A terminal result is not valid until the real service-stop receipt and cleanup trace close with no managed resource remaining.
- A competing writer, lost lease, timeout, cancellation, output flood, probe exhaustion, manifest conflict, stale parent hash, uncommitted verdict, old attempt, or invalid proof graph is always a structured non-pass.
- Model-authored plans use the strict proposal and ToolSpec contracts. Before dispatch, require current `agent-context.json`, a passed current plan audit, and `action-contracts.json`; every real action must have a durable intent and matching commit in `action-journal.jsonl`. Only ToolSpec-declared idempotent unresolved intents may be retried with the same deterministic key. Read `references/action-protocol.md`.
- State events, stage/action/cleanup trace spans, command hashes, immutable attempt, verdict, and repository-context hash must agree. Currentness is content/hash based; do not rely on mtime, filename, or an old context snapshot.
- `scripts/agent_critic_cli.py` and `scripts/agent_schedule_cli.py` produce proposal/scheduling advice only. Their `not_authorization=true` output—including a parallel batch—cannot bypass the deterministic runtime policy boundary.
- `scripts/human_control_cli.py` records Ed25519-verified HITL and confirmed knowledge; it never executes an approved action. Production HITL/Knowledge journals additionally require an unexpired checkpoint signed by an external checkpoint authority, while local-test mode is explicitly non-production. Read `references/human-control.md`.
- `scripts/agent_slo_report.py --run-dir` accepts only proof-verified run roots for production SLO qualification. Trace-only reports are analysis and are always non-production.
- `scripts/agent_eval.py --production` is only an evaluation gate backed by a signed evaluator registration and always declares `not_authorization=true`. Production scoring requires a public-key-allowlisted evaluator registration that binds the frozen held-out corpus, candidate, deterministic baseline, thresholds, and proof-backed SLO input set. P2 parallel/multi-agent release additionally requires `scripts/agent_release_admission.py` to recompute both reports; the derived admission is not itself a signed attestation. Read `references/agent-release-admission.md`.
- A P2 release admission is not runtime tool authorization and does not automatically enable parallel or multi-Agent execution. Real production qualification requires the externally controlled signed corpus/authorities and proof-backed runs; bundled fixtures cannot supply it.
- `--skip-probe` is handoff-only: it dispatches no actions, issues no action contracts, and can never unlock `PASS`.

## Evidence Integrity Contract

Be strict. Do not optimize for a nice-looking report; optimize for truth.

- Do not fabricate requirements, data, results, screenshots, response bodies, logs, timings, defect counts, or coverage percentages.
- Do not mark a requirement as passed unless the run produced direct evidence for that requirement.
- Do not mark a requirement as `Passed` while any mapped test remains `Failed`, `Blocked`, `Untested`, or `Inconclusive`.
- Do not infer backend/data correctness from a successful UI render alone. Verify API, persistence, logs, or returned data when the requirement depends on data flow.
- Do not hide untested areas. If something was not tested, mark it `Untested`. If it could not be tested, mark it `Blocked` and state the blocker.
- Do not collapse multiple requirement points into one generic pass. Each explicit acceptance criterion needs its own status.
- Do not treat mock/demo data as real data unless the requirement explicitly accepts mock data.
- Do not use stale screenshots, unreadable placeholder images, old report data, or previous-run results as current evidence unless the report labels them as historical context. Passed requirements must cite evidence marked `current_run: true`; screenshot evidence must point to a readable PNG/JPEG image; when `results.json.startedAt` exists, current-run file evidence should be generated during or after that run.
- Do not cite evidence for a `Passed` requirement or test when the evidence lineage belongs to another requirement or test. Runner-generated evidence should carry `requirement_ids` and `test_ids`; if lineage exists, it must match the cited item. When `results.json` is available, evidence generated by `ledger_from_probe.py` must also bind to a matching current results step by scenario, step id/action, lineage, and status, and copied fields such as status code, checked JSON, matched text, message counts, runtime counts, headers, and artifact paths must be preserved and match the bound step; do not delete copied runner fields and replace them with free-text assertions.
- Do not accept generic success text as current-run proof when stale seed data, cached responses, or front-end fallback text could match. Use the runner's generated `qa_marker` and assert that the returned stream/API/persistence evidence contains it when the feature can echo or persist user-provided content.
- Do not treat ledger text-match fields as self-proving. Matched or forbidden response, message, request body, stdout, or stderr text must reference a captured artifact file where the matched text is present and forbidden text is absent.
- Do not treat `checked_json`, `checked_request_json`, or `checked_stdout_json` as self-proving. They must reference a captured response, request, stream-message, or stdout artifact that parses as JSON and contains the same checked path/value pairs.
- Do not treat `extracted_json`, `extracted_stdout_json`, checked response headers, or extracted response headers as self-proving. Extracted ids, turn ids, job ids, trace ids, or statuses must match their recorded source path/header, and response header checks/extractions must include the captured `response_headers` map.
- Do not treat hand-written stream `assertions`, zero-message counts, or a bare/missing `messages_path` string as captured WebSocket/SSE message evidence. Passed stream tests need a message count greater than zero, an existing readable non-empty current-run message artifact path, matched returned message text, or checked/extracted stream-message JSON evidence.
- Do not treat hand-written `proves` or generic `assertions` text as terminal-status proof. Claims such as `answer_done`, `completed`, or terminal state need returned message text, checked response JSON/text, checked stdout JSON/text, or extracted returned/output status evidence, including for API and UI-to-API same-object reads.
- Do not treat runtime disposition fields as self-proving. If `results.json` contains console errors, failed responses, or request failures, `checked_* = 0` only proves disposition when the matching `ignored_*` count accounts for every observed runtime issue of that category; otherwise re-run a focused runtime disposition probe after the observed issue or report the issue.
- Keep evidence layers separate. UI visibility or fallback text, test seed setup, stream terminal events, same-object API reads, and persistence/log terminal state are different proof layers. Do not merge them into one pass claim.
- Do not treat `business-model.json`, `oracle-model.json`, `qa-metrics.json`, or `closeout-candidates.json` as proof. They are planning, oracle, measurement, and human-confirmation handoff artifacts; final pass still requires current-run evidence and `qa-verdict.json`.
- If evidence is ambiguous, write `Inconclusive`, explain why, and name the exact evidence still needed.
- Do not claim final pass from `audit-summary.json` alone. The audit can prove ledger structure while `qa-verdict.json` still correctly returns `failed`, `blocked`, or `inconclusive`.
- Do not mix artifacts across runs. `qa-verdict.json` must be generated from an `audit-summary.json` whose ledger/results paths, content hashes, and referenced evidence artifact hashes match the current `evidence-ledger.json`, `results.json`, and evidence files.
- Do not let `results.json.artifactDir` point to another run directory. Relative runner artifacts, copied evidence fields, and report defaults are current only when `artifactDir` matches the current ledger/results artifact directory.
- Do not generate final pass from an audit that omitted `test-matrix.json`; matrix coverage and ledger completeness must be part of the audited artifact set.
- Do not omit `--results` when `audit-summary.json` was generated with `results.json`; otherwise runtime errors, failed requests, and request failures are no longer bound to the final verdict.
- Do not omit existing sibling `defects.json`, `requirement-coverage.json`, `plan-audit-summary.json`, `service-preflight.json`, `service-runtime.json`, `adapter-probes.json`, `adapter-context.json`, or `qa-cycle-error.json` from final verdict generation; known defects, source coverage gaps, invalid plans, setup blockers, adapter blockers, environment/data boundaries, or QA pipeline failures must not be hidden by leaving artifacts out.
- Do not substitute a same-named artifact from another run for an existing current-run sibling artifact. If `defects.json`, `requirement-coverage.json`, `plan-audit-summary.json`, `service-preflight.json`, `service-runtime.json`, `adapter-probes.json`, `adapter-context.json`, or `qa-cycle-error.json` exists beside the current ledger/results, verdict and report generation must use that current artifact or block the pass claim.
- Do not run probes when `requirement-coverage.json` says requirement-source units are unmapped, unless the user explicitly wants a planning/blocker report. `--allow-unmapped-requirement-source` may allow execution to continue so blocker artifacts can be produced, but coverage remains incomplete and no pass claim is allowed.
- Treat unconfirmed runtime/data boundary as a first-class blocker. If `qa-verdict.json` contains both environment-boundary reason codes and other non-pass reasons such as defects, runtime gaps, or strategy gaps, confirm the environment/data boundary before reporting a product conclusion or auto-continuing follow-up probes.

Allowed statuses:

- `Passed`: directly verified with evidence.
- `Failed`: tested and contradicted expected behavior.
- `Blocked`: could not be tested because of a concrete blocker.
- `Untested`: not reached within the run scope.
- `Inconclusive`: evidence exists but is insufficient or contradictory.

## Requirement Intake

When the user provides an issue, PR, URL, screenshot, or free-form requirement:

- Gather the latest requirement text from the provided source. If the source is a GitHub issue/PR, inspect the description, comments that change scope, linked commits if relevant, and changed files when useful.
- If the user writes the requirement directly, use that text as the source of truth.
- If the current browser page is relevant, inspect the visible UI and network behavior before deciding the test surface.
- Do not ask for clarification unless a destructive action, production mutation, or missing credential blocks meaningful testing.
- For a first-pass artifact, run `scripts/init_qa_artifact.py` or `scripts/scaffold_requirement.py` with the requirement source. Treat the generated charter, matrix, and plan as a conservative scaffold to review, not as proof that the requirement is fully testable.
- When a project checkout is available, generate `adapter-context.json` with `scripts/discover_project_context.py` or through `scripts/init_qa_artifact.py --project-root <repo>`. Use it to state environment/data boundaries and route evidence to the right service, stream, persistence, or log layer.
- For a real backtest pass claim, set `--runtime-mode` and `--data-boundary-status` when initializing or running the cycle. Environment/data confirmation is required by default; `--allow-unconfirmed-environment` is an explicit partial-run exception and must not be described as a real-environment pass.

Extract these fields into a working test charter:

- User goal: what outcome the feature should produce.
- Actors and permissions: who can do it and who should not.
- Entry points: pages, buttons, commands, API endpoints, scheduled jobs, or background flows.
- Data flow: required input, API calls, persistence, derived data, displayed data, and downstream effects.
- Logic rules: validation, branching, ordering, state transitions, authorization, retries, idempotency.
- Interaction rules: loading, disabled states, modal behavior, keyboard/mouse basics, responsive behavior, toasts, empty/error states.
- Acceptance criteria: explicit criteria from the requirement plus implicit criteria needed for the feature to work end to end.
- Business model: actors, entities, workflows, state transitions, rules, entry points, and agent-team handoff boundaries.
- Oracle model: requirement-specific pass rules, required evidence layers, weak signals to avoid, and blocked-until inputs.
- Risks and unknowns: parts that cannot be safely verified yet.

Use `references/test-charter.md` when a structured charter template is needed.

Completeness rule: after extraction, compare the test matrix back to the original requirement source line by line or paragraph by paragraph. Any requirement source line that implies behavior, data, permission, or interaction must map to at least one test item or to a documented `Out of scope`, `Blocked`, or `Untested` note.

## Planning

Create a dynamic test matrix before running broad tests. Prefer a compact table with:

- Requirement point
- Test type: logic, interaction, data/API, permission, error, regression, responsive
- Steps or probe
- Expected result
- Evidence to capture
- Status

For repeatable browser checks, create a JSON plan compatible with `scripts/playwright_probe.mjs`. Read `references/plan-schema.md` before authoring or editing the plan.

For strict evidence ledgers, read `references/evidence-ledger-schema.md` before deciding final statuses.

Do not include stale page lists just because a previous run used them. Add a route, API, or interaction only when the current requirement or discovered dependency justifies it.

Before execution, perform a coverage check:

- The target environment, data boundary, service status, and adapter assumptions are recorded in `adapter-context.json` or explicitly documented in the charter.
- Every explicit requirement has at least one planned test.
- Every important data dependency has an API/log/persistence verification method or a documented blocker.
- Every create/update/delete flow has a safe test-data strategy.
- Every user-visible workflow has an interaction check for loading, disabled, validation, success, and error states when applicable.
- Every planned assertion names the evidence that will prove or disprove it.

## Execution

Use the lightest reliable evidence for each claim:

- Browser/UI: Playwright, in-app browser, or Chrome when login state is required.
- API/data flow: direct HTTP probes, application logs, database reads, or existing project test helpers. Prefer helper commands that print JSON and assert them with `expectStdoutJson` / `extractStdoutJson` instead of loose stdout text checks when verifying persistence state. Use response-header assertions or `extractResponseHeader` when trace ids, cache/content-type, auth-dependent headers, or gateway routing are part of the proof.
- Logic: combine UI behavior with API/state evidence rather than relying on visual checks alone.
- Interaction: verify click targets with `expectClickable` before important clicks, use `clickAndWaitForResponse` when a click must prove API/data continuity, plus form validation, loading/disabled states, modal lifecycle, navigation, toasts, empty states, and responsive breakpoints.
- UI blockers: handle optional onboarding, modals, masks, and locale/copy variants explicitly with planned dismiss/assertion steps. Treat visible-but-not-clickable controls as a separate interaction failure with hit-test blocker evidence, not as a visual pass.
- Errors: capture console errors, failed requests, HTTP 4xx/5xx, unhandled exceptions, traceback/log snippets, and user-visible error states.
- Cross-step state: when a stream/API creates a session, turn, job, upload, or trace id, use `extractJson`, `extractResponseHeader`, and later `{ "var": "..." }` references or `pathTemplate` placeholders such as `/api/items/{id}` so follow-up API, command, log, or persistence probes verify the same object. For current-run proof, inject `qa_marker` with `{ "var": "qa_marker" }` or `{ "template": "..." }`, enable `captureRequestBody` when the request payload itself is part of the claim, assert request-side payload with `expectRequestTextContains` or `expectRequestJson`, and assert the same marker appears in the returned API, stream, log, or persistence layer. Use `captureResponseHeaders` only when the header set itself is evidence. For asynchronous jobs or sessions, use `pollApi` to repeat the same-object read until the terminal status assertion passes or the poll timeout is reached. For authorized create/update/delete flows in a test environment, include a safe test-data cleanup strategy; use `cleanupApi` with `alwaysRun` and `skipIfMissingVars` when a created runtime id must be removed after assertions.
- Failure diagnostics: for WebSocket/SSE/API flows, prefer plans that can keep running after a terminal error when diagnostic identifiers are available. Use scenario-level `continueOnFailure` with downstream checks to distinguish stream failure, session persistence, and database status instead of collapsing them into one vague failure.
- Layer gates: `stream`/`websocket`/`sse` tests that pass must cite WebSocket/SSE message evidence; `api` and `ui_to_api` tests must cite API/click-to-response evidence; `persistence` tests must cite persistence/log/API evidence. Claims about current-run markers, stale seed avoidance, fallback avoidance, `answer_done`, or `completed` need returned marker or terminal-status evidence, not only request text or screenshots. Returned marker evidence must come from returned message/response/stdout text or checked returned JSON that contains the actual marker value; request-body marker evidence alone does not count.

Default artifact layout:

```text
<out-dir>/<timestamp>-<slug>/
├── requirement.md
├── business-model.json
├── oracle-model.json
├── qa-metrics.json
├── closeout-candidates.json
├── semantic-artifacts-summary.json
├── adapter-context.json
├── adapter-probes.json
├── service-preflight.json
├── service-runtime.json
├── test-charter.md
├── test-matrix.json
├── test-plan.json
├── scaffold-summary.json
├── requirement-coverage.json
├── plan-audit-summary.json
├── results.json
├── evidence-ledger.json
├── audit-summary.json
├── defects.json
├── next-probes.json
├── next-probe-application.json
├── qa-cycle-error.json
├── qa-verdict.json
├── qa-agent-summary.json
├── qa-agent-handoff.md
├── qa-run-summary.json
├── screenshots/
├── evidence/
└── report.md
```

Helpful scripts:

- `scripts/init_qa_artifact.py`: create a run folder and scaffold requirement, business model, oracle model, charter, matrix, plan, metrics, closeout candidates, summary, and initial ledger files. Missing, unreadable, or directory-shaped requirement input files, plus unreadable project roots discovered while generating adapter context, create a blocked run folder with `qa-initialization-error.json`, `scaffold-summary.json` `input_artifact_errors`, and a blocked initial ledger, then exit non-zero instead of crashing or fabricating probes from invalid scope.
- `scripts/discover_project_context.py`: inspect a checkout for adapter/environment context, service ports, config boundaries, package scripts, and evidence-layer warnings without reading secret values. Generic root-level services use stable ids such as `node-app`, `python-service`, or `go-service` instead of `.` so required-service preflight checks can target them precisely. Missing, file-shaped, or unreadable project roots are written to `adapter-context.json` as `input_artifact_errors` with `project_root_status.readable=false` and exit non-zero instead of silently producing a generic no-service context.
- `scripts/preflight_runtime.py`: verify required service ports, start-command executables, npm scripts, Node dependencies for required npm/pnpm/yarn services, env/config file paths, and plan-declared required files/directories before execution. It writes `service-preflight.json` and never reads secret values or starts services by default. During `--refresh-context`, preserve custom services from the existing `adapter-context.json` and re-probe their readiness so post-start checks verify the same service ids. Explicit `--required-service` ids missing from adapter context are blockers. Missing, unreadable, directory-shaped, malformed, or non-object adapter-context/plan inputs, plus unreadable project roots discovered during refresh, are written to `service-preflight.json` as `input_artifact_errors` and exit non-zero instead of crashing or synthesizing a start plan.
- `scripts/service_runtime.py`: dry-run or explicitly start missing local/test services from `service-preflight.json` `start_plan`. It writes `service-runtime.json` with PID, command, cwd, stdout/stderr log paths, and port-readiness evidence. Omit `--start` for a dry-run; use `--stop` only for PIDs recorded by the same runtime artifact. Missing, unreadable, directory-shaped, malformed, or non-object preflight/runtime inputs are written to service runtime artifacts as `input_artifact_errors` and exit non-zero without starting or stopping services. `--stop` also exits non-zero whenever any recorded process cannot be safely identified or terminated; callers must inspect `summary.failed_count` and may not treat a written cleanup report as cleanup success.
- `scripts/synthesize_adapter_probes.py`: convert adapter context plus matrix/plan into executable adapter probes or explicit blockers when a matched `references/adapters/*.json` definition exposes a safe probe template. Project markers, services, ports, env/config candidates, and probe defaults belong to adapter JSON, not generic scripts. Missing, unreadable, directory-shaped, malformed, or non-object adapter-context/plan/matrix inputs are written to `adapter-probes.json` as `input_artifact_errors`; `--apply` does not rewrite plan/matrix when inputs are invalid.
- `scripts/scaffold_requirement.py`: derive a conservative first-pass `business-model.json`, `oracle-model.json`, `test-charter.md`, `test-matrix.json`, `test-plan.json`, `qa-metrics.json`, `closeout-candidates.json`, and `scaffold-summary.json` from requirement text. It generates runnable UI/read-only API probes when entry points are explicit, adds `expectClickable` interaction probes for inferable button/click requirements, generates `clickAndWaitForResponse` UI-to-API probes when the same requirement names a click target and API path, can extract ids from an authorized click response and verify a same-object read-only follow-up API via `pathTemplate` or `pollApi` when asynchronous terminal status is mentioned, adds `cleanupApi` teardown probes for authorized create flows with same-object ids, models CLI/command requirements as blocked command-oracle rows until project root, runtime/data boundary, command safety, and expected stdout/stderr assertions are confirmed, models responsive/mobile/desktop, disabled-state validation, forbidden-request, list query/sort/pagination, empty-state, retryable-error, stale-data-guard, upload/file-fixture, multipart request, request-marker, file-validation, file-security scan/quarantine, secure file preview, progress, async-poll, download/export artifact, async report artifact-generation/resume/cancel/partial-failure/download-guard, response-header, CSV content, PII redaction, webhook signature, idempotency/replay, notification/outbox, notification-preference policy, no-real-email dry-run, tenant/data isolation, workspace-boundary denial, cross-tenant denial, forbidden foreign-tenant/text absence, no-persistence side-effect, timezone-aware date ranges, inclusive start, exclusive end, DST boundary, money precision, rounding, tax/discount, currency conversion, persisted calculation parity, keyboard/focus/ARIA accessibility, bulk selection/destructive-confirmation/soft-delete/undo, concurrency/atomicity/locking/conflict/no-oversell, background-job/worker/retry-backoff/dead-letter/alert-outbox, feature-flag/rollout/cohort/default-off/stale-flag, realtime collaboration/broadcast/multi-client/order/reconnect-replay, CSRF/session/cookie-security, OAuth/OIDC PKCE callback, redirect-uri allowlist/open-redirect denial, SAML/SSO ACS request/assertion/signature/audience/recipient/replay security, WebAuthn/passkey challenge/assertion/attestation/sign-count security, MFA/TOTP/recovery-code security, password-reset/magic-link/email-verification one-time-token security, API-key/PAT secret-once/hash-only/scope/expiry/last-used/revocation security, audit-log append-only/hash-chain/tamper-denial/retention/legal-hold security, privacy export/erasure DSAR artifact/legal-hold/session-key-revocation/search-cache-purge security, GraphQL/BFF persisted-query/field-authorization/resolver-batching/mutation/subscription-replay security, RAG grounded-answer/retrieval-trace/citation-span/source-excerpt/document-version/prompt-injection/abstention security, usage-quota metering/counter/reset-boundary security, and rate-limit/lockout/no-session/account-enumeration requirements as separate oracle layers instead of folding them into a generic screenshot, status code, queued response, beta-user success, single stream message, generic artifact job queued response, generic artifact download success, generic 403, generic 409, generic 429, generic privacy export queued response, generic erasure API success, generic GraphQL HTTP 200 or partial-error response, generic RAG answer text, generic retrieval success, generic stream completion, generic OAuth callback 200, generic redirect, generic SAML/SSO ACS success, generic WebAuthn/passkey login success, generic MFA challenge success, generic password-reset/email-sent success, generic API-key create/list/use success, generic audit-log row visibility, generic quota API 200/409, upload success, download success, delete-button click, or clickability check, and marks stream, persistence, permission, placeholder, unlocatable click/edit targets, unsafe click-to-response mutations, bulk selected-row fixtures, destructive confirmation/cancel evidence, soft-delete/undo operation-id evidence, upload/download fixtures, file scan/quarantine fixtures, signed preview/header/no-leak evidence, webhook signature fixtures, artifact job/progress/manifest/hash/resume/cancel/partial-failure/download-guard fixtures, privacy DSAR/export/erasure/legal-hold fixtures, GraphQL operation/variables/persisted-query, FIELD_DENIED/partial-data, resolver_trace/DataLoader, mutation side-effect, subscription cursor/replay fixtures, RAG request/answer stream/retrieval trace/vector index/source citation/citation span/source excerpt/document version/prompt-injection/abstention fixtures, idempotency replay fixtures, notification worker/outbox evidence, notification preference/consent/suppression/quiet-hours/digest/unsubscribe-token fixtures, background worker/queue evidence, retry/dead-letter fault-injection evidence, feature-flag evaluation evidence, rollout cohort/control fixtures, realtime multi-client auth/fixture evidence, reconnect cursor replay evidence, CSRF token/header/denial fixtures, session rotation/logout invalidation fixtures, Set-Cookie flag assertions, OAuth state/nonce/PKCE/code-exchange/replay-denial fixtures, redirect_uri/return_to allowlist and open-redirect denial fixtures, SAML AuthnRequest/SAMLRequest/RelayState/ACS/SP entityID fixtures, signed and negative SAMLResponse fixtures, XML signature/x509, audience/recipient/InResponseTo/time-window, NameID/group mapping, request-consumption/replay-denial, no-session side-effect, and secret-redaction fixtures, WebAuthn challenge/options, rpId/origin/clientDataJSON/authenticatorData, signature/user-verification/sign-count, replay/wrong-origin/wrong-rpId/unknown-credential denial, attestation/public-key storage, and secret-redaction fixtures, MFA challenge/TOTP/recovery-code fixtures, one-time token hash/purpose/expiry/consumption/replay fixtures, reset/verification email outbox and link evidence, existing-vs-unknown account no-enumeration comparison, password/session side-effect checks, API-key admin/auth fixture, env-backed secret material, secret-once/hash/prefix/scope/expiry/list proof, scoped allow/deny proof, last_used_at/revoked_at proof, revoked/expired/tampered denial, no-secret-leak assertions, audit event fixture, sequence/hash-chain recomputation, append-only write proof, tamper-denial/no-mutation proof, retention/legal-hold proof, pseudonym/PII-redaction proof, privacy export artifact/manifest/data_hash proof, erasure pseudonymization, active-session deletion, API-key revocation, search-index removal, cache purge, legal-hold block proof, quota tenant/meter/window fixtures, usage-counter and counter-version proof, concurrent quota winner/loser requests, quota-exceeded no-side-effect proof, billing-event exactness, quota reset-boundary worker/audit evidence, rate-limit threshold/retry-after/lockout-reset/no-session/enumeration fixtures, stale-flag guards, fault-injection error states, cross-tenant/workspace fixture gaps, time-boundary fixture gaps, calculation fixture gaps, accessibility selector/focus fixture gaps, concurrency contention fixture gaps, or mutating checks as blocked until the needed auth, payload, helper, selector, file fixture, failure fixture, downloaded artifact, artifact worker/manifest/resume/cancel/download-guard fixture, scan-status poll, signed preview token, response headers, selected ids/count, confirmation text, undo window, safe recipient boundary, queue/worker log, notification preference/consent/suppression/quiet-hours/digest/unsubscribe-token fixtures, privacy DSAR/erasure fixtures, GraphQL BFF fixtures, operationName/variables/persisted-query capture, field-denied partial-data proof, resolver batching/N+1 proof, subscription replay/cursor proof, RAG knowledge fixtures, same answer_id stream terminal event, retrieval_trace/query_hash/vector_index capture, retrieved source ids, citation spans with source excerpt matches, current document_version/stale-source denial, prompt-injection safety_trace, tenant/corpus isolation, insufficient-sources abstention/no-citation-row proof, retry/dead-letter fixture, flag evaluation/config version, beta/control/anonymous fixtures, sender/recipient client contexts, ordered sequence capture, current/old session cookies, OAuth IDP fixture, state/nonce/PKCE verifier, code-exchange capture, redirect allowlist fixture, SAML IdP fixture, AuthnRequest/SAMLRequest and RelayState capture, ACS/SP binding, signed/negative assertion fixtures, XML-signature/x509 proof, audience/recipient/InResponseTo/time-window proof, NameID/group mapping proof, request-consumption proof, WebAuthn/passkey fixture, challenge/options capture, rpId/origin assertion data, public-key signature verification, sign-count counter evidence, attestation/public-key storage proof, MFA account/challenge/code fixture, one-time-token reset/link fixture, token hash/purpose/expiry/consumption proof, outbox/link proof, replay/expired/tampered/wrong-purpose/wrong-tenant denial, password/session side-effect proof, API-key secret-once/hash/prefix/scope/expiry/list proof, scoped allow/deny proof, last_used_at/revoked_at transitions, revoked/expired/tampered denial, no-secret-leak assertions, audit event fixture, sequence/hash-chain recomputation helper, append-only/tamper-denial attempts, retention/legal-hold state, pseudonym/PII-redaction checks, privacy export/erasure legal-hold proof, quota tenant/meter/window fixture, atomic usage-counter transition, counter-version evidence, idempotent duplicate absence, quota-exceeded denial side-effect absence, billing usage event exactness, reset-boundary worker/audit proof, rate-limit account/client key, attempt counter, lockout expiry, Retry-After header, no-session proof, account-enumeration comparison, reference calculation, simultaneous request orchestration, or safe test data is supplied. Code-review PR text with source/config/test paths is scaffolded as `code_pr` command evidence and those file paths are not treated as browser/API routes. Missing, unreadable, or directory-shaped requirement inputs are written to `scaffold-summary.json` as `input_artifact_errors` with blocked matrix/ledger-compatible artifacts and no product probes.
- Search relevance is modeled as its own oracle family. Requirements that mention search ranking, `relevance_score`, `ranking_model`, query rewrite/canonical query, typo tolerance, synonyms, facet counts, sponsored disclosure, pagination duplicate absence, stale-result fallback, or tenant-hidden products must preserve those as `search_relevance`, `result_order`, `result_position`, `query_rewrite`, `canonical_query`, `facet_counts`, `total_count`, `sponsored_disclosure`, `stale_result_guard`, and data-isolation evidence layers; a generic search page screenshot, list API 200, sort check, or stale-error message is not enough.
- Scheduled jobs are modeled as their own oracle family. Requirements that mention cron/schedule expressions, `schedule_id`, `run_key`, `next_run_at`, missed-run catch-up, due windows, timezone/DST boundaries, scheduler/advisory locks, `already_running`/`duplicate_skipped`, dry-run side-effect absence, generated rows, outbox/email boundaries, or scheduler audit rows must preserve `scheduled_job`, `schedule_expression`, `scheduler_run`, `run_key`, `due_window`, `catch_up`, `scheduler_lock`, `dry_run`, `invoice_rows`, duplicate-absence, no-real-email, persistence, and runtime evidence; a generic command exit 0, queued job row, worker log, or API/list blocker is not enough.
- Cleanup is modeled as its own oracle layer when a requirement creates or mutates safe test data and then requires teardown. Plans must preserve the extracted runtime id, same-object readback, `cleanupApi` or equivalent cleanup command, always-run teardown behavior, deleted/absent verification, related-row cleanup, outbox absence, audit evidence such as `testdata_deleted`, persistence, and runtime disposition; a generic delete button click, a best-effort teardown note, or a cleanup `proves` string is not enough.
- Decision-table and business-logic routing requirements are their own oracle shape, not a generic command/API/UI check. Plans must preserve the named fixture rows, stdout JSON, `expected_decisions`, rule-hit matrix, rule precedence/override rows, boundary rows, negative rows, expected outputs for every branch, terminal status, and runtime disposition; command exit 0, UI screenshots, or API 200 alone are not sufficient.
- `scripts/modeling_benchmark.py`: run built-in UI/API/stream/CLI/responsive-validation/list-query/upload-async/download-export/webhook-notification-idempotency/notification-preference-policy/webhook-signature-replay-window/tenant-data-isolation/time-boundary/money-precision/accessibility-modal-keyboard/decision-table-logic/checkout-localization-locale-fallback/concurrency-no-oversell/background-job-retry-dead-letter/scheduled-job-timezone-dedupe/feature-flag-rollout-default-off/realtime-collaboration-broadcast-reconnect/csrf-session-cookie-security/login-rate-limit-lockout/bulk-soft-delete-undo-audit/project-create-same-object-cleanup/attachment-malware-preview-security/oauth-pkce-callback-security/mfa-to

…(truncated)