Automated QA Test

自动化 QA-Test

Use this skill as a dynamic QA harness. Do not hardcode product pages, route lists, business rules, or expected results inside the skill. Derive them each time from the current request, issue, PR, requirement text, repo behavior, and visible runtime evidence.

Core Principle

Treat every run as a fresh requirement audit:

1. Read the requirement source.
2. Extract expected behavior, actors, state changes, data dependencies, permissions, edge cases, and success/failure states.
3. Turn those into a test charter and executable test plan.
4. Test feature logic, interaction quality, data/API continuity, and runtime errors.
5. Report what was covered, what passed, what failed, and what remains unverified.

Agent Loop

Use this skill as an evidence-producing agent loop, not as a static checklist:

1. Compile the requirement into business-model.json, oracle-model.json, test-charter.md, test-matrix.json, test-plan.json, qa-metrics.json, and closeout-candidates.json. For a new free-form requirement, start with scripts/init_qa_artifact.py or scripts/scaffold_requirement.py so the first draft already maps business actors/entities/workflows to UI/API/stream/persistence probes or explicit blockers. Code-review PR text with source/config/test file paths belongs in code_pr command-check mode unless the requirement also names explicit runtime UI/API acceptance criteria.
2. Select probes for each matrix row: UI, API, WebSocket/SSE, command/log/persistence, or manual blocker. When adapter-context.json exists, run scripts/synthesize_adapter_probes.py or run_qa_cycle.py --synthesize-adapter-probes before validation to convert known safe adapter paths into executable probes.
3. Run scripts/preflight_runtime.py or run_qa_cycle.py --preflight-runtime when local services, ports, or project tooling affect execution. Treat preflight blockers as run blockers, not test failures. If the target is a local/test checkout and the user wants the agent to bring services up, run scripts/service_runtime.py --start or run_qa_cycle.py --start-missing-services so startup PID/log/readiness evidence is captured in service-runtime.json.
4. Run scripts/audit_requirement_coverage.py or the default run_qa_cycle.py coverage gate when requirement.md exists. Every requirement-source behavior point must map to test-matrix.json before probes run.
5. Run scripts/validate_plan.py to catch unmapped matrix requirements, weak probe steps, TODO placeholders, secret-like values, and risky commands before execution.
6. Run scripts/run_qa_cycle.py for the full executable loop when using the bundled probe runner.
7. Apply safe concrete follow-up recommendations with scripts/apply_next_probes.py or run_qa_cycle.py --apply-next-probes before the next execution cycle.
8. Manually update evidence-ledger.json only when custom probes outside the runner were used.
9. Generate qa-verdict.json with scripts/generate_verdict.py or through run_qa_cycle.py. Treat can_claim_pass=false as the guardrail against calling a failed, blocked, untested, inconclusive, or unaudited run a pass.
10. Generate or write the report only from the audited ledger, verdict, and explicitly labeled gaps.

If the feature spans multiple services or has repo-specific data ownership rules, read references/project-adapters.md before planning probes.

When changing or interpreting business-model.json, oracle-model.json, qa-metrics.json, or closeout-candidates.json, read references/business-oracle-model.md.

Proof-Carrying Control Boundary

• Treat qa-run-summary.json, reports, model text, semantic models, and memory as projections or context, never as authority for PASS.
• A normal cycle must hold the run lease, execute through the shared RunBudget, commit current outputs to an immutable attempt, append the deterministic terminal state, and pass scripts/verify_run_proof.py.
• The outer qa_agent_loop.py owns one budget and lease across all iterations. Probe/output consumption never resets between iterations, and a child cycle may inherit only the exact live parent lease.
• Once a managed service is started, ordinary work must preserve the cycle's cleanup deadline/output reserve. A terminal result is not valid until the real service-stop receipt and cleanup trace close with no managed resource remaining.
• A competing writer, lost lease, timeout, cancellation, output flood, probe exhaustion, manifest conflict, stale parent hash, uncommitted verdict, old attempt, or invalid proof graph is always a structured non-pass.
• Model-authored plans use the strict proposal and ToolSpec contracts. Before dispatch, require current agent-context.json, a passed current plan audit, and action-contracts.json; every real action must have a durable intent and matching commit in action-journal.jsonl. Only ToolSpec-declared idempotent unresolved intents may be retried with the same deterministic key. Read references/action-protocol.md.
• State events, stage/action/cleanup trace spans, command hashes, immutable attempt, verdict, and repository-context hash must agree. Currentness is content/hash based; do not rely on mtime, filename, or an old context snapshot.
• scripts/agent_critic_cli.py and scripts/agent_schedule_cli.py produce proposal/scheduling advice only. Their not_authorization=true output—including a parallel batch—cannot bypass the deterministic runtime policy boundary.
• scripts/human_control_cli.py records Ed25519-verified HITL and confirmed knowledge; it never executes an approved action. Production HITL/Knowledge journals additionally require an unexpired checkpoint signed by an external checkpoint authority, while local-test mode is explicitly non-production. Read references/human-control.md.
• scripts/agent_slo_report.py --run-dir accepts only proof-verified run roots for production SLO qualification. Trace-only reports are analysis and are always non-production.
• scripts/agent_eval.py --production is only an evaluation gate backed by a signed evaluator registration and always declares not_authorization=true. Production scoring requires a public-key-allowlisted evaluator registration that binds the frozen held-out corpus, candidate, deterministic baseline, thresholds, and proof-backed SLO input set. P2 parallel/multi-agent release additionally requires scripts/agent_release_admission.py to recompute both reports; the derived admission is not itself a signed attestation. Read references/agent-release-admission.md.
• A P2 release admission is not runtime tool authorization and does not automatically enable parallel or multi-Agent execution. Real production qualification requires the externally controlled signed corpus/authorities and proof-backed runs; bundled fixtures cannot supply it.
• --skip-probe is handoff-only: it dispatches no actions, issues no action contracts, and can never unlock PASS.

Evidence Integrity Contract

Be strict. Do not optimize for a nice-looking report; optimize for truth.

• Do not fabricate requirements, data, results, screenshots, response bodies, logs, timings, defect counts, or coverage percentages.
• Do not mark a requirement as passed unless the run produced direct evidence for that requirement.
• Do not mark a requirement as Passed while any mapped test remains Failed, Blocked, Untested, or Inconclusive.
• Do not infer backend/data correctness from a successful UI render alone. Verify API, persistence, logs, or returned data when the requirement depends on data flow.
• Do not hide untested areas. If something was not tested, mark it Untested. If it could not be tested, mark it Blocked and state the blocker.
• Do not collapse multiple requirement points into one generic pass. Each explicit acceptance criterion needs its own status.
• Do not treat mock/demo data as real data unless the requirement explicitly accepts mock data.
• Do not use stale screenshots, unreadable placeholder images, old report data, or previous-run results as current evidence unless the report labels them as historical context. Passed requirements must cite evidence marked current_run: true; screenshot evidence must point to a readable PNG/JPEG image; when results.json.startedAt exists, current-run file evidence should be generated during or after that run.
• Do not cite evidence for a Passed requirement or test when the evidence lineage belongs to another requirement or test. Runner-generated evidence should carry requirement_ids and test_ids; if lineage exists, it must match the cited item. When results.json is available, evidence generated by ledger_from_probe.py must also bind to a matching current results step by scenario, step id/action, lineage, and status, and copied fields such as status code, checked JSON, matched text, message counts, runtime counts, headers, and artifact paths must be preserved and match the bound step; do not delete copied runner fields and replace them with free-text assertions.
• Do not accept generic success text as current-run proof when stale seed data, cached responses, or front-end fallback text could match. Use the runner's generated qa_marker and assert that the returned stream/API/persistence evidence contains it when the feature can echo or persist user-provided content.
• Do not treat ledger text-match fields as self-proving. Matched or forbidden response, message, request body, stdout, or stderr text must reference a captured artifact file where the matched text is present and forbidden text is absent.
• Do not treat checked_json, checked_request_json, or checked_stdout_json as self-proving. They must reference a captured response, request, stream-message, or stdout artifact that parses as JSON and contains the same checked path/value pairs.
• Do not treat extracted_json, extracted_stdout_json, checked response headers, or extracted response headers as self-proving. Extracted ids, turn ids, job ids, trace ids, or statuses must match their recorded source path/header, and response header checks/extractions must include the captured response_headers map.
• Do not treat hand-written stream assertions, zero-message counts, or a bare/missing messages_path string as captured WebSocket/SSE message evidence. Passed stream tests need a message count greater than zero, an existing readable non-empty current-run message artifact path, matched returned message text, or checked/extracted stream-message JSON evidence.
• Do not treat hand-written proves or generic assertions text as terminal-status proof. Claims such as answer_done, completed, or terminal state need returned message text, checked response JSON/text, checked stdout JSON/text, or extracted returned/output status evidence, including for API and UI-to-API same-object reads.
• Do not treat runtime disposition fields as self-proving. If results.json contains console errors, failed responses, or request failures, checked_* = 0 only proves disposition when the matching ignored_* count accounts for every observed runtime issue of that category; otherwise re-run a focused runtime disposition probe after the observed issue or report the issue.
• Keep evidence layers separate. UI visibility or fallback text, test seed setup, stream terminal events, same-object API reads, and persistence/log terminal state are different proof layers. Do not merge them into one pass claim.
• Do not treat business-model.json, oracle-model.json, qa-metrics.json, or closeout-candidates.json as proof. They are planning, oracle, measurement, and human-confirmation handoff artifacts; final pass still requires current-run evidence and qa-verdict.json.
• If evidence is ambiguous, write Inconclusive, explain why, and name the exact evidence still needed.
• Do not claim final pass from audit-summary.json alone. The audit can prove ledger structure while qa-verdict.json still correctly returns failed, blocked, or inconclusive.
• Do not mix artifacts across runs. qa-verdict.json must be generated from an audit-summary.json whose ledger/results paths, content hashes, and referenced evidence artifact hashes match the current evidence-ledger.json, results.json, and evidence files.
• Do not let results.json.artifactDir point to another run directory. Relative runner artifacts, copied evidence fields, and report defaults are current only when artifactDir matches the current ledger/results artifact directory.
• Do not generate final pass from an audit that omitted test-matrix.json; matrix coverage and ledger completeness must be part of the audited artifact set.
• Do not omit --results when audit-summary.json was generated with results.json; otherwise runtime errors, failed requests, and request failures are no longer bound to the final verdict.
• Do not omit existing sibling defects.json, requirement-coverage.json, plan-audit-summary.json, service-preflight.json, service-runtime.json, adapter-probes.json, adapter-context.json, or qa-cycle-error.json from final verdict generation; known defects, source coverage gaps, invalid plans, setup blockers, adapter blockers, environment/data boundaries, or QA pipeline failures must not be hidden by leaving artifacts out.
• Do not substitute a same-named artifact from another run for an existing current-run sibling artifact. If defects.json, requirement-coverage.json, plan-audit-summary.json, service-preflight.json, service-runtime.json, adapter-probes.json, adapter-context.json, or qa-cycle-error.json exists beside the current ledger/results, verdict and report generation must use that current artifact or block the pass claim.
• Do not run probes when requirement-coverage.json says requirement-source units are unmapped, unless the user explicitly wants a planning/blocker report. --allow-unmapped-requirement-source may allow execution to continue so blocker artifacts can be produced, but coverage remains incomplete and no pass claim is allowed.
• Treat unconfirmed runtime/data boundary as a first-class blocker. If qa-verdict.json contains both environment-boundary reason codes and other non-pass reasons such as defects, runtime gaps, or strategy gaps, confirm the environment/data boundary before reporting a product conclusion or auto-continuing follow-up probes.

Allowed statuses:

• Passed: directly verified with evidence.
• Failed: tested and contradicted expected behavior.
• Blocked: could not be tested because of a concrete blocker.
• Untested: not reached within the run scope.
• Inconclusive: evidence exists but is insufficient or contradictory.

Requirement Intake

When the user provides an issue, PR, URL, screenshot, or free-form requirement:

• Gather the latest requirement text from the provided source. If the source is a GitHub issue/PR, inspect the description, comments that change scope, linked commits if relevant, and changed files when useful.
• If the user writes the requirement directly, use that text as the source of truth.
• If the current browser page is relevant, inspect the visible UI and network behavior before deciding the test surface.
• Do not ask for clarification unless a destructive action, production mutation, or missing credential blocks meaningful testing.
• For a first-pass artifact, run scripts/init_qa_artifact.py or scripts/scaffold_requirement.py with the requirement source. Treat the generated charter, matrix, and plan as a conservative scaffold to review, not as proof that the requirement is fully testable.
• When a project checkout is available, generate adapter-context.json with scripts/discover_project_context.py or through scripts/init_qa_artifact.py --project-root <repo>. Use it to state environment/data boundaries and route evidence to the right service, stream, persistence, or log layer.
• For a real backtest pass claim, set --runtime-mode and --data-boundary-status when initializing or running the cycle. Environment/data confirmation is required by default; --allow-unconfirmed-environment is an explicit partial-run exception and must not be described as a real-environment pass.

Extract these fields into a working test charter:

• User goal: what outcome the feature should produce.
• Actors and permissions: who can do it and who should not.
• Entry points: pages, buttons, commands, API endpoints, scheduled jobs, or background flows.
• Data flow: required input, API calls, persistence, derived data, displayed data, and downstream effects.
• Logic rules: validation, branching, ordering, state transitions, authorization, retries, idempotency.
• Interaction rules: loading, disabled states, modal behavior, keyboard/mouse basics, responsive behavior, toasts, empty/error states.
• Acceptance criteria: explicit criteria from the requirement plus implicit criteria needed for the feature to work end to end.
• Business model: actors, entities, workflows, state transitions, rules, entry points, and agent-team handoff boundaries.
• Oracle model: requirement-specific pass rules, required evidence layers, weak signals to avoid, and blocked-until inputs.
• Risks and unknowns: parts that cannot be safely verified yet.

Use references/test-charter.md when a structured charter template is needed.

Completeness rule: after extraction, compare the test matrix back to the original requirement source line by line or paragraph by paragraph. Any requirement source line that implies behavior, data, permission, or interaction must map to at least one test item or to a documented Out of scope, Blocked, or Untested note.

Planning

Create a dynamic test matrix before running broad tests. Prefer a compact table with:

• Requirement point
• Test type: logic, interaction, data/API, permission, error, regression, responsive
• Steps or probe
• Expected result
• Evidence to capture
• Status

For repeatable browser checks, create a JSON plan compatible with scripts/playwright_probe.mjs. Read references/plan-schema.md before authoring or editing the plan.

For strict evidence ledgers, read references/evidence-ledger-schema.md before deciding final statuses.

Do not include stale page lists just because a previous run used them. Add a route, API, or interaction only when the current requirement or discovered dependency justifies it.

Before execution, perform a coverage check:

• The target environment, data boundary, service status, and adapter assumptions are recorded in adapter-context.json or explicitly documented in the charter.
• Every explicit requirement has at least one planned test.
• Every important data dependency has an API/log/persistence verification method or a documented blocker.
• Every create/update/delete flow has a safe test-data strategy.
• Every user-visible workflow has an interaction check for loading, disabled, validation, success, and error states when applicable.
• Every planned assertion names the evidence that will prove or disprove it.

Execution

Use the lightest reliable evidence for each claim:

• Browser/UI: Playwright, in-app browser, or Chrome when login state is required.
• API/data flow: direct HTTP probes, application logs, database reads, or existing project test helpers. Prefer helper commands that print JSON and assert them with expectStdoutJson / extractStdoutJson instead of loose stdout text checks when verifying persistence state. Use response-header assertions or extractResponseHeader when trace ids, cache/content-type, auth-dependent headers, or gateway routing are part of the proof.
• Logic: combine UI behavior with API/state evidence rather than relying on visual checks alone.
• Interaction: verify click targets with expectClickable before important clicks, use clickAndWaitForResponse when a click must prove API/data continuity, plus form validation, loading/disabled states, modal lifecycle, navigation, toasts, empty states, and responsive breakpoints.
• UI blockers: handle optional onboarding, modals, masks, and locale/copy variants explicitly with planned dismiss/assertion steps. Treat visible-but-not-clickable controls as a separate interaction failure with hit-test blocker evidence, not as a visual pass.
• Errors: capture console errors, failed requests, HTTP 4xx/5xx, unhandled exceptions, traceback/log snippets, and user-visible error states.
• Cross-step state: when a stream/API creates a session, turn, job, upload, or trace id, use extractJson, extractResponseHeader, and later { "var": "..." } references or pathTemplate placeholders such as /api/items/{id} so follow-up API, command, log, or persistence probes verify the same object. For current-run proof, inject qa_marker with { "var": "qa_marker" } or { "template": "..." }, enable captureRequestBody when the request payload itself is part of the claim, assert request-side payload with expectRequestTextContains or expectRequestJson, and assert the same marker appears in the returned API, stream, log, or persistence layer. Use captureResponseHeaders only when the header set itself is evidence. For asynchronous jobs or sessions, use pollApi to repeat the same-object read until the terminal status assertion passes or the poll timeout is reached. For authorized create/update/delete flows in a test environment, include a safe test-data cleanup strategy; use cleanupApi with alwaysRun and skipIfMissingVars when a created runtime id must be removed after assertions.
• Failure diagnostics: for WebSocket/SSE/API flows, prefer plans that can keep running after a terminal error when diagnostic identifiers are available. Use scenario-level continueOnFailure with downstream checks to distinguish stream failure, session persistence, and database status instead of collapsing them into one vague failure.
• Layer gates: stream/websocket/sse tests that pass must cite WebSocket/SSE message evidence; api and ui_to_api tests must cite API/click-to-response evidence; persistence tests must cite persistence/log/API evidence. Claims about current-run markers, stale seed avoidance, fallback avoidance, answer_done, or completed need returned marker or terminal-status evidence, not only request text or screenshots. Returned marker evidence must come from returned message/response/stdout text or checked returned JSON that contains the actual marker value; request-body marker evidence alone does not count.

Default artifact layout:

<out-dir>/<timestamp>-<slug>/
├── requirement.md
├── business-model.json
├── oracle-model.json
├── qa-metrics.json
├── closeout-candidates.json
├── semantic-artifacts-summary.json
├── adapter-context.json
├── adapter-probes.json
├── service-preflight.json
├── service-runtime.json
├── test-charter.md
├── test-matrix.json
├── test-plan.json
├── scaffold-summary.json
├── requirement-coverage.json
├── plan-audit-summary.json
├── results.json
├── evidence-ledger.json
├── audit-summary.json
├── defects.json
├── next-probes.json
├── next-probe-application.json
├── qa-cycle-error.json
├── qa-verdict.json
├── qa-agent-summary.json
├── qa-agent-handoff.md
├── qa-run-summary.json
├── screenshots/
├── evidence/
└── report.md

Helpful scripts:

• scripts/init_qa_artifact.py: create a run folder and scaffold requirement, business model, oracle model, charter, matrix, plan, metrics, closeout candidates, summary, and initial ledger files. Missing, unreadable, or directory-shaped requirement input files, plus unreadable project roots discovered while generating adapter context, create a blocked run folder with qa-initialization-error.json, scaffold-summary.json input_artifact_errors, and a blocked initial ledger, then exit non-zero instead of crashing or fabricating probes from invalid scope.
• scripts/discover_project_context.py: inspect a checkout for adapter/environment context, service ports, config boundaries, package scripts, and evidence-layer warnings without reading secret values. Generic root-level services use stable ids such as node-app, python-service, or go-service instead of . so required-service preflight checks can target them precisely. Missing, file-shaped, or unreadable project roots are written to adapter-context.json as input_artifact_errors with project_root_status.readable=false and exit non-zero instead of silently producing a generic no-service context.
• scripts/preflight_runtime.py: verify required service ports, start-command executables, npm scripts, Node dependencies for required npm/pnpm/yarn services, env/config file paths, and plan-declared required files/directories before execution. It writes service-preflight.json and never reads secret values or starts services by default. During --refresh-context, preserve custom services from the existing adapter-context.json and re-probe their readiness so post-start checks verify the same service ids. Explicit --required-service ids missing from adapter context are blockers. Missing, unreadable, directory-shaped, malformed, or non-object adapter-context/plan inputs, plus unreadable project roots discovered during refresh, are written to service-preflight.json as input_artifact_errors and exit non-zero instead of crashing or synthesizing a start plan.
• scripts/service_runtime.py: dry-run or explicitly start missing local/test services from service-preflight.json start_plan. It writes service-runtime.json with PID, command, cwd, stdout/stderr log paths, and port-readiness evidence. Omit --start for a dry-run; use --stop only for PIDs recorded by the same runtime artifact. Missing, unreadable, directory-shaped, malformed, or non-object preflight/runtime inputs are written to service runtime artifacts as input_artifact_errors and exit non-zero without starting or stopping services. --stop also exits non-zero whenever any recorded process cannot be safely identified or terminated; callers must inspect summary.failed_count and may not treat a written cleanup report as cleanup success.
• scripts/synthesize_adapter_probes.py: convert adapter context plus matrix/plan into executable adapter probes or explicit blockers when a matched references/adapters/*.json definition exposes a safe probe template. Project markers, services, ports, env/config candidates, and probe defaults belong to adapter JSON, not generic scripts. Missing, unreadable, directory-shaped, malformed, or non-object adapter-context/plan/matrix inputs are written to adapter-probes.json as input_artifact_errors; --apply does not rewrite plan/matrix when inputs are invalid.
• scripts/scaffold_requirement.py: derive a conservative first-pass business-model.json, oracle-model.json, test-charter.md, test-matrix.json, test-plan.json, qa-metrics.json, closeout-candidates.json, and scaffold-summary.json from requirement text. It generates runnable UI/read-only API probes when entry points are explicit, adds expectClickable interaction probes for inferable button/click requirements, generates clickAndWaitForResponse UI-to-API probes when the same requirement names a click target and API path, can extract ids from an authorized click response and verify a same-object read-only follow-up API via pathTemplate or pollApi when asynchronous terminal status is mentioned, adds cleanupApi teardown probes for authorized create flows with same-object ids, models CLI/command requirements as blocked command-oracle rows until project root, runtime/data boundary, command safety, and expected stdout/stderr assertions are confirmed, models responsive/mobile/desktop, disabled-state validation, forbidden-request, list query/sort/pagination, empty-state, retryable-error, stale-data-guard, upload/file-fixture, multipart request, request-marker, file-validation, file-security scan/quarantine, secure file preview, progress, async-poll, download/export artifact, async report artifact-generation/resume/cancel/partial-failure/download-guard, response-header, CSV content, PII redaction, webhook signature, idempotency/replay, notification/outbox, notification-preference policy, no-real-email dry-run, tenant/data isolation, workspace-boundary denial, cross-tenant denial, forbidden foreign-tenant/text absence, no-persistence side-effect, timezone-aware date ranges, inclusive start, exclusive end, DST boundary, money precision, rounding, tax/discount, currency conversion, persisted calculation parity, keyboard/focus/ARIA accessibility, bulk selection/destructive-confirmation/soft-delete/undo, concurrency/atomicity/locking/conflict/no-oversell, background-job/worker/retry-backoff/dead-letter/alert-outbox, feature-flag/rollout/cohort/default-off/stale-flag, realtime collaboration/broadcast/multi-client/order/reconnect-replay, CSRF/session/cookie-security, OAuth/OIDC PKCE callback, redirect-uri allowlist/open-redirect denial, SAML/SSO ACS request/assertion/signature/audience/recipient/replay security, WebAuthn/passkey challenge/assertion/attestation/sign-count security, MFA/TOTP/recovery-code security, password-reset/magic-link/email-verification one-time-token security, API-key/PAT secret-once/hash-only/scope/expiry/last-used/revocation security, audit-log append-only/hash-chain/tamper-denial/retention/legal-hold security, privacy export/erasure DSAR artifact/legal-hold/session-key-revocation/search-cache-purge security, GraphQL/BFF persisted-query/field-authorization/resolver-batching/mutation/subscription-replay security, RAG grounded-answer/retrieval-trace/citation-span/source-excerpt/document-version/prompt-injection/abstention security, usage-quota metering/counter/reset-boundary security, and rate-limit/lockout/no-session/account-enumeration requirements as separate oracle layers instead of folding them into a generic screenshot, status code, queued response, beta-user success, single stream message, generic artifact job queued response, generic artifact download success, generic 403, generic 409, generic 429, generic privacy export queued response, generic erasure API success, generic GraphQL HTTP 200 or partial-error response, generic RAG answer text, generic retrieval success, generic stream completion, generic OAuth callback 200, generic redirect, generic SAML/SSO ACS success, generic WebAuthn/passkey login success, generic MFA challenge success, generic password-reset/email-sent success, generic API-key create/list/use success, generic audit-log row visibility, generic quota API 200/409, upload success, download success, delete-button click, or clickability check, and marks stream, persistence, permission, placeholder, unlocatable click/edit targets, unsafe click-to-response mutations, bulk selected-row fixtures, destructive confirmation/cancel evidence, soft-delete/undo operation-id evidence, upload/download fixtures, file scan/quarantine fixtures, signed preview/header/no-leak evidence, webhook signature fixtures, artifact job/progress/manifest/hash/resume/cancel/partial-failure/download-guard fixtures, privacy DSAR/export/erasure/legal-hold fixtures, GraphQL operation/variables/persisted-query, FIELD_DENIED/partial-data, resolver_trace/DataLoader, mutation side-effect, subscription cursor/replay fixtures, RAG request/answer stream/retrieval trace/vector index/source citation/citation span/source excerpt/document version/prompt-injection/abstention fixtures, idempotency replay fixtures, notification worker/outbox evidence, notification preference/consent/suppression/quiet-hours/digest/unsubscribe-token fixtures, background worker/queue evidence, retry/dead-letter fault-injection evidence, feature-flag evaluation evidence, rollout cohort/control fixtures, realtime multi-client auth/fixture evidence, reconnect cursor replay evidence, CSRF token/header/denial fixtures, session rotation/logout invalidation fixtures, Set-Cookie flag assertions, OAuth state/nonce/PKCE/code-exchange/replay-denial fixtures, redirect_uri/return_to allowlist and open-redirect denial fixtures, SAML AuthnRequest/SAMLRequest/RelayState/ACS/SP entityID fixtures, signed and negative SAMLResponse fixtures, XML signature/x509, audience/recipient/InResponseTo/time-window, NameID/group mapping, request-consumption/replay-denial, no-session side-effect, and secret-redaction fixtures, WebAuthn challenge/options, rpId/origin/clientDataJSON/authenticatorData, signature/user-verification/sign-count, replay/wrong-origin/wrong-rpId/unknown-credential denial, attestation/public-key storage, and secret-redaction fixtures, MFA challenge/TOTP/recovery-code fixtures, one-time token hash/purpose/expiry/consumption/replay fixtures, reset/verification email outbox and link evidence, existing-vs-unknown account no-enumeration comparison, password/session side-effect checks, API-key admin/auth fixture, env-backed secret material, secret-once/hash/prefix/scope/expiry/list proof, scoped allow/deny proof, last_used_at/revoked_at proof, revoked/expired/tampered denial, no-secret-leak assertions, audit event fixture, sequence/hash-chain recomputation, append-only write proof, tamper-denial/no-mutation proof, retention/legal-hold proof, pseudonym/PII-redaction proof, privacy export artifact/manifest/data_hash proof, erasure pseudonymization, active-session deletion, API-key revocation, search-index removal, cache purge, legal-hold block proof, quota tenant/meter/window fixtures, usage-counter and counter-version proof, concurrent quota winner/loser requests, quota-exceeded no-side-effect proof, billing-event exactness, quota reset-boundary worker/audit evidence, rate-limit threshold/retry-after/lockout-reset/no-session/enumeration fixtures, stale-flag guards, fault-injection error states, cross-tenant/workspace fixture gaps, time-boundary fixture gaps, calculation fixture gaps, accessibility selector/focus fixture gaps, concurrency contention fixture gaps, or mutating checks as blocked until the needed auth, payload, helper, selector, file fixture, failure fixture, downloaded artifact, artifact worker/manifest/resume/cancel/download-guard fixture, scan-status poll, signed preview token, response headers, selected ids/count, confirmation text, undo window, safe recipient boundary, queue/worker log, notification preference/consent/suppression/quiet-hours/digest/unsubscribe-token fixtures, privacy DSAR/erasure fixtures, GraphQL BFF fixtures, operationName/variables/persisted-query capture, field-denied partial-data proof, resolver batching/N+1 proof, subscription replay/cursor proof, RAG knowledge fixtures, same answer_id stream terminal event, retrieval_trace/query_hash/vector_index capture, retrieved source ids, citation spans with source excerpt matches, current document_version/stale-source denial, prompt-injection safety_trace, tenant/corpus isolation, insufficient-sources abstention/no-citation-row proof, retry/dead-letter fixture, flag evaluation/config version, beta/control/anonymous fixtures, sender/recipient client contexts, ordered sequence capture, current/old session cookies, OAuth IDP fixture, state/nonce/PKCE verifier, code-exchange capture, redirect allowlist fixture, SAML IdP fixture, AuthnRequest/SAMLRequest and RelayState capture, ACS/SP binding, signed/negative assertion fixtures, XML-signature/x509 proof, audience/recipient/InResponseTo/time-window proof, NameID/group mapping proof, request-consumption proof, WebAuthn/passkey fixture, challenge/options capture, rpId/origin assertion data, public-key signature verification, sign-count counter evidence, attestation/public-key storage proof, MFA account/challenge/code fixture, one-time-token reset/link fixture, token hash/purpose/expiry/consumption proof, outbox/link proof, replay/expired/tampered/wrong-purpose/wrong-tenant denial, password/session side-effect proof, API-key secret-once/hash/prefix/scope/expiry/list proof, scoped allow/deny proof, last_used_at/revoked_at transitions, revoked/expired/tampered denial, no-secret-leak assertions, audit event fixture, sequence/hash-chain recomputation helper, append-only/tamper-denial attempts, retention/legal-hold state, pseudonym/PII-redaction checks, privacy export/erasure legal-hold proof, quota tenant/meter/window fixture, atomic usage-counter transition, counter-version evidence, idempotent duplicate absence, quota-exceeded denial side-effect absence, billing usage event exactness, reset-boundary worker/audit proof, rate-limit account/client key, attempt counter, lockout expiry, Retry-After header, no-session proof, account-enumeration comparison, reference calculation, simultaneous request orchestration, or safe test data is supplied. Code-review PR text with source/config/test paths is scaffolded as code_pr command evidence and those file paths are not treated as browser/API routes. Missing, unreadable, or directory-shaped requirement inputs are written to scaffold-summary.json as input_artifact_errors with blocked matrix/ledger-compatible artifacts and no product probes.
• Search relevance is modeled as its own oracle family. Requirements that mention search ranking, relevance_score, ranking_model, query rewrite/canonical query, typo tolerance, synonyms, facet counts, sponsored disclosure, pagination duplicate absence, stale-result fallback, or tenant-hidden products must preserve those as search_relevance, result_order, result_position, query_rewrite, canonical_query, facet_counts, total_count, sponsored_disclosure, stale_result_guard, and data-isolation evidence layers; a generic search page screenshot, list API 200, sort check, or stale-error message is not enough.
• Scheduled jobs are modeled as their own oracle family. Requirements that mention cron/schedule expressions, schedule_id, run_key, next_run_at, missed-run catch-up, due windows, timezone/DST boundaries, scheduler/advisory locks, already_running/duplicate_skipped, dry-run side-effect absence, generated rows, outbox/email boundaries, or scheduler audit rows must preserve scheduled_job, schedule_expression, scheduler_run, run_key, due_window, catch_up, scheduler_lock, dry_run, invoice_rows, duplicate-absence, no-real-email, persistence, and runtime evidence; a generic command exit 0, queued job row, worker log, or API/list blocker is not enough.
• Cleanup is modeled as its own oracle layer when a requirement creates or mutates safe test data and then requires teardown. Plans must preserve the extracted runtime id, same-object readback, cleanupApi or equivalent cleanup command, always-run teardown behavior, deleted/absent verification, related-row cleanup, outbox absence, audit evidence such as testdata_deleted, persistence, and runtime disposition; a generic delete button click, a best-effort teardown note, or a cleanup proves string is not enough.
• Decision-table and business-logic routing requirements are their own oracle shape, not a generic command/API/UI check. Plans must preserve the named fixture rows, stdout JSON, expected_decisions, rule-hit matrix, rule precedence/override rows, boundary rows, negative rows, expected outputs for every branch, terminal status, and runtime disposition; command exit 0, UI screenshots, or API 200 alone are not sufficient.
• scripts/modeling_benchmark.py: run built-in UI/API/stream/CLI/responsive-validation/list-query/upload-async/download-export/webhook-notification-idempotency/notification-preference-policy/webhook-signature-replay-window/tenant-data-isolation/time-boundary/money-precision/accessibility-modal-keyboard/decision-table-logic/checkout-localization-locale-fallback/concurrency-no-oversell/background-job-retry-dead-letter/scheduled-job-timezone-dedupe/feature-flag-rollout-default-off/realtime-collaboration-broadcast-reconnect/csrf-session-cookie-security/login-rate-limit-lockout/bulk-soft-delete-undo-audit/project-create-same-object-cleanup/attachment-malware-preview-security/oauth-pkce-callback-security/mfa-to

…(truncated)