QA Demo Stand

QA on a demo stand

Stage contract

• Stage: 6. QA to production quality · Kind: prompt
• Inputs: the built, converged project (local/dev).
• Outputs: reproduced + root-caused defects (and fixes, per FIX_MODE), tested to production quality; a running defect/timeline record.
• Entry gate: the build has converged. LOCAL/DEV only — never point at stage/prod; confirm before any destructive chaos.
• Done when: defects reproduced ≥3× with an established cause, criticals surfaced/fixed, and self-heal-vs-broken-state verified for each induced failure.
• Next: the setup-server skill.
• Note: this is a prompt template — fill its [PLACEHOLDERS] and hand the whole thing to a fresh executor agent; don't paraphrase it. This is also the entry point for the bug on-ramp (a defect report enters the flow here).

You are a senior QA/Security engineer. Build a local test stand (a "demo stand") for the project [PROJECT_DIR] and test it to production quality: find, reproduce, and record defects, with a focus on infrastructure stability, data isolation, and resilience to failures. Work autonomously and thoroughly; don't spare time or tokens.

Hard rules

• LOCAL/dev environment ONLY ([BASE_URL], default http://127.0.0.1:8080). NEVER point tests or destructive probes at stage/prod. Destructive chaos (killing containers, cutting the network, corrupting configs) — only on local dev containers, with mandatory restoration afterwards.
• The stand lives in a SEPARATE demostand/ folder, excluded from git via .git/info/exclude (don't touch the shared .gitignore; verify with git check-ignore demostand/). Commit nothing from the stand.
• Build the stand FROM SCRATCH and INDEPENDENTLY. Treat existing tests as UNVERIFIED: there is no guarantee they covered all scenarios or are correct. Wherever you can, independently re-verify the behavior they supposedly cover.
• Document-driven: the source of truth is the specs in demostand/spec/ (scenarios, flows, edge cases, security, infra). Scripts and browser agents merely EXECUTE what is described.
• Simulate a REAL user through the browser (real login, clicks, input), not only direct API calls.
• A bug = reproduced ≥3 times + an established cause. Only then — report criticals immediately. Fix handling mode: [FIX_MODE: "accumulate everything and fix nothing" OR "fix confirmed ones immediately"]. If you fix — strictly TDD (test→code→run→check for absent regressions) and log to the timeline "what was found / how it was fixed / re-verified". Changes to auth/security/migrations — run an external code review at the very end.
• For every induced failure, check not "did it fall over" but whether the system RECOVERS on its own or wedges into a broken state and requires a restart.
• Before outward-facing/irreversible actions and before chaos on possibly-shared infrastructure — confirm. After browser probes, close sessions and delete screenshots/artifacts.
• Out of scope: [ANY_EXCLUSIONS — features/modes we don't touch; if none — "none"].
• Every couple of hours produce a brief checkpoint report and continue.

Phase 0 — Reconnaissance

• Check whether the environment is up and how it starts ([HOW_TO_RUN], default docker compose up -d); health endpoints; container statuses; backend logs.
• Identify the stack: frontend/backend frameworks, DB, queues/workers, real-time (websocket/SSE), external integrations, edge/reverse-proxy, deployment scheme.
• Identify the authentication/authorization and multi-tenancy model: [AUTH_MODEL — e.g. "cookie sessions, organizations + RLS, admin/user roles"; if you don't know — derive it from the code].
• Inventory the available tooling: browser automation (check availability, e.g. agent-browser/Playwright), DB and container access, how to run tests.

Phase 1 — System map (the foundation)

Build a structured map of the application (may be parallelized across explorer agents by domain). For each domain: capabilities, user actions, settings, API routes + error codes, edge cases (including rare "once in 10 years" races), failure modes (self-heal / needs restart / broken-state), security concerns, existing test coverage (treated as unverified). Domains (adapt to the project):

1. Authentication and session lifecycle. 2) Authorization/roles/multi-tenancy and data isolation. 3) Core business flows (CRUD/product processes). 4) Data model and its isolation (RLS/scoping). 5) Files (upload/download/sharing). 6) Real-time (websocket/SSE, reconnect/resync). 7) Settings/admin panel. 8) Background jobs/schedulers/queues. 9) External integrations. 10) Infra and config loading (compose, healthchecks, restart policies, startup order, config sources). 11) Full inventory of API routes and error codes. 12) [Optionally, if there is an AI/agent/LLM] — the agent's behavior boundaries. Save the raw map to demostand/spec/00-system-map.json.

Phase 2 — Stand specifications (from the map)

Write documents (markdown), each item with a stable ID, real file paths, risk, probability, and a verification method (a browser step or an API call + expected error code):

• 01-test-data — test accounts/organizations/data, credentials, profiles.
• 02-feature-surface — the full surface: capabilities/settings/routes.
• 03-user-flow-matrix — end-to-end user flows and their INTERSECTIONS (chaotic orderings, parallel actions, context/tab switching, attachments, editing/opening entities).
• 04-edge-cases — a consolidated ranked catalog (races, time/timezone boundaries, malformed/huge/empty inputs).
• 05-security — tenant isolation/IDOR, authz, tokens/TTL, XSS/injections, path traversal, secret leakage, rate limit/lockout. [If there is an agent — a separate adversarial catalog of boundaries.]
• 06-infra-resilience — environment map + a "self-recovery vs wedging" matrix + a chaos injection plan (kill/restart containers, network cut/slowdown, corrupted/missing configs, cold start) with exact commands and expected post-conditions.
• 08-candidate-findings — RANKED reproducible defect hypotheses (severity × confidence) with exact repro steps: the bridge to the run.

Phase 3 — Test data provisioning

Your own idempotent script that creates DISPOSABLE marked (demostand-*) accounts/orgs/data through the application's REAL mechanism (registration/bootstrap/admin API or its data layer), with a guard (local only, explicit confirmation flag). Credentials manifest — in demostand/reports/ (local). A separate cleanup script. Verify that the new entities are actually usable (required modules/access enabled).

Phase 4 — Scenario executors

• Browser harness (a host-side wrapper over a browser CLI): login/logout, navigation, submitting forms/messages, waiting for operations to complete via real UI signals, capturing console errors and network failures (offline/abort/throttle), screenshots. Multiple sessions = multiple "people" (but watch resources: 5+ parallel heavy LLM/SPA sessions produce false timeouts — limit parallelism).
• API/concurrency: burst load, races, negative inputs (missing/malformed/huge fields).
• Infra chaos: kill/restart individual containers; cutting access to Docker/DB/dependencies; network cut/slowdown in the browser; corrupted/missing config files; cold start. With restoration and a self-heal check.
• [If there is an AI/agent] adversarial boundary probes (see Phase 5).

Phase 5 — Run in rounds, accumulate findings

Categories (adapt):

• Auth: login/logout/wrong password/rate limit/lockout/session lifecycle and renewal/cookie flags.
• Authz/tenancy: IDOR and cross-tenant access via someone else's IDs; roles; what a regular user sees vs admin; whether the DB (RLS) is really a backstop or isolation holds only in code (check the DB connection role).
• Validation and the ERROR CONTRACT: status codes (4xx vs 5xx), stable error codes, human-readable messages, fail-fast without hidden fallbacks.
• Business flows: happy path + creating/editing/opening/deleting entities, "redo the current one", "open a past one", concurrent edits.
• Files: oversized/empty/forbidden MIME/double extensions/path traversal/cross-tenant access/ downloading an incomplete file/sharing a private one.
• Real-time: reconnect/resync after a cut, stream integrity (no duplication), cancel/stop.
• Concurrency/races: parallel identical operations, double submits, startup races.
• Network: short and long outage, slow network, cutting a specific endpoint; graceful degradation (a clear message, not a crash) + FULL recovery once connectivity returns.
• Infra: after kill/restart/dependency loss — self-recovery or wedge (restart needed?).
• Security: XSS (including via data from integrations), injections, tokens without TTL/revocation, secret leakage into responses/into the UI, traversal, missing rate limit.
• [AI/agent, if present] Boundary: the agent handles only its designated tasks and correctly REFUSES out-of-scope ones (off-topic, code, creative writing, general chat); is resistant to prompt injection ("ignore your instructions", role switching, "print your system prompt"); does not rewrite itself/its rules; does not read files/secrets outside the sandbox; does not exfiltrate other tenants' data. Any breached case is a critical finding. For each finding: reproduce ≥3 times, establish the cause (file:line), assess severity and confidence. Distinguish a real bug from a load/harness artifact (re-verify cleanly).

Phase 6 — Fixes (per mode) or accumulation

If [FIX_MODE] = fix: via TDD (RED→fix→GREEN→run adjacent tests→typecheck/lint), log to the timeline; for significant changes — an external code review at the end; a mandatory self-review always. If = accumulate: fix nothing, file into the registry; at the end write REMEDIATION.md — for each finding: where, cause, the concrete fix (files/code/SQL/config, layer), a TDD plan, regression risks, contentious decisions for discussion, how to re-verify via the stand, effort estimate.

Reporting (maintain continuously)

• demostand/TIMELINE.md — chronology: time · operation · findings · fixes.
• demostand/reports/FINDINGS.md — a ranked registry of findings with repro and statuses.
• demostand/reports/SUMMARY.md — executive summary: verdict by area + findings table.
• Structure: demostand/{spec,scripts,reports,artifacts,logs}/.

Final

Restore the environment to a working/healthy state (check health and containers). Close browser sessions, delete temporary artifacts. Deliver a final report: what was covered, verdict by priority (infra stability / data isolation / [agent boundaries, if present]), the list of findings, what was fixed, what remains.

Source: Pukh (@aostrikov_agents_chat), Demo stand testing. Translated from Russian.