Playtest Scenarios

Playtest Scenarios: Author and Replay

Mission

An application's user-facing surface — commands, flags, arguments, menus, buttons, endpoints, file formats — is a contract. This skill turns that contract into a standing library of scenario cards, then replays the library unchanged whenever either side of the contract may have moved:

• the app shipped a feature, refactor, or dependency bump, or
• the runner changed — a new model, model version, or agent harness is now driving the app.

Because every pass runs the same cards with the same exact inputs, drift shows up as a status change on a specific card instead of a surprise in production. The library also acts as a trust gate: a new model is not relied on for real deliverables from a repository until it has completed the pass, and until more than one distinct runner has completed it.

This is exploratory, user-facing playtesting with a repeatable baseline. It is not unit testing and not a static code audit.

The two parts

Part 1 — Author. Scan the repository, inventory its user-facing interface, and create the scenario library at <target-repo>/docs/test_scenarios/. Trigger: no library exists yet, or a deliberate amendment is requested after confirmed app drift.

Part 2 — Replay ("the check"). Execute the library's cards verbatim, record a dated run report, compare against prior runs, classify drift, and issue a qualification verdict for the runner. Trigger: a new model or model version, a changed agent harness, an app release or dependency change, a head-to-head comparison of models on this repository, or an acceptance pass before shipping a deliverable produced with this repository.

If Part 2 is requested and no library exists, run Part 1 first in the same session, say so explicitly, and label the resulting report a baseline run rather than a comparison. Never silently substitute an ad hoc scenario list for an existing library.

Library contract (shared by both parts)

Location: <target-repo>/docs/test_scenarios/. Resolve this against the target repository, never against the skill's own directory or the caller's shell directory.

docs/test_scenarios/
  README.md          # scope, safety rails, run order, pass shapes, coverage
  01-<topic>.md      # numbered card files in dependency order
  02-<topic>.md
  runs/              # dated run reports, append-only
    2026-07-25-<runner>.md

Binding rules:

• Cards are standing artifacts. A replay never edits, regenerates, normalizes, or appends to them. Results go in the run report only.
• Card edits happen only in a deliberate Part 1 authoring or amendment pass, with the library version bumped in the README.
• The only target-repo mutations this skill permits: Part 1 creates or amends files inside docs/test_scenarios/; Part 2 adds one new report under docs/test_scenarios/runs/ and may update the README's qualified-runners table. Never modify application source, configuration, or unrelated docs.
• Run reports are append-only history. Never rewrite an old report.
• Adopting an existing library. A library authored outside this skill is still authoritative. Follow its documented card format, order, safety limits, and execution rules as written. If it lacks a runs/ directory or a qualified-runners table, a replay may create the directory and append the missing table to the README — clearly labeled, without touching any card or any other README section. Report format gaps (missing coverage checklist, no pass shapes) as library findings; never rewrite the library to match this skill's templates.
• Report routing under adoption. When the adopted library routes reports to its own destination (an audits directory, a session log, a defect ledger), honor that routing too — but the per-card run record still goes under runs/, because cross-run comparison depends on an append-only history in one place. If the library's destination is untracked by version control (a local-only archive), runs/ holds the full durable report and the library's destination gets a pointer; otherwise a compact runs/ entry may link the full report. Confirmed defects additionally go wherever the library's contract files them.

Execution contract (shared by both parts)

These rules make results comparable across runs and honest about what was observed. Part 1 writes a tailored version of them into the library README; Part 2 honors the library's version and falls back to these defaults where the library is silent.

1. Record the baseline. Every run captures: app version or commit, OS/architecture, surface under test, runner identity, environment root, and scenario ID before results are recorded.
2. Start from declared state. Unless a card names a dependency on an earlier card, give it a clean disposable environment and fresh fixtures. Record any reused state. Never let an earlier card's leftovers create an accidental pass.
3. Self-report is not evidence. When the target app embeds a model or agent, that model's claim about its own identity, working directory, tool call, or saved file proves nothing. Corroborate every such claim with filesystem state, protocol capture, UI metadata, exit codes, or logs.
4. Test each surface separately. When a card names multiple surfaces (CLI, TUI, desktop, API), record a separate result per surface. Success on one surface does not imply success on another.
5. Apply deadlines. Use the library's declared response deadlines; where it has none, apply sensible defaults (local feedback within seconds, not minutes) and record elapsed time. Environment-caused slowness may be Blocked; an unexplained hang or missed cancellation is Fail.
6. Assert observable outcomes. Exit codes, stdout/stderr separately, before/after file state for persistence cards, process/port state for lifecycle cards. Redact credentials before attaching evidence.
7. Pass atomically. Every statement under a card's Expected section is an assertion; if one fails, the card fails. Variations are separately labeled subcases.
8. Preserve failure artifacts. On failure, stop mutating the fixture until logs, config, exact input, and relevant files are captured. Retry from a cloned fixture, never by repairing the evidence in place.

Part 1 — Author the library

Repo discovery

Before writing any card, learn what the application is:

1. App type — web app, desktop app, CLI, API service, TUI, plugin, data-processing job, multi-service system. Signals: README.md, package.json, pyproject.toml, Cargo.toml, go.mod, docker-compose.yml, Makefile, .env.example, and folders like src/, app/, routes/, components/, cmd/, tests/, examples/.
2. Run method — the repo-native launch command from docs or scripts. Prefer the lowest-risk local development mode when several exist.
3. Inputs — every user-facing input: CLI commands, subcommands, flags, positional arguments, environment variables, config keys, text fields, dropdowns, file uploads, keyboard shortcuts, API request bodies.
4. Outputs — screens, tables, generated files, exports, exit codes, error messages, logs, persisted settings, database records.
5. Workflows — create, edit, save, load, delete, import, export, search, configure, cancel midway, relaunch, recover.
6. Persistence and failure modes — where state lives and what likely breaks.

Interface inventory

This step is what makes replays comparable. Record the exact surface: every command and flag with exact spelling, argument types and defaults, environment variables, config keys, named screens and controls, API routes, and file formats. Cards must quote these exactly. A card that says "run the export command" cannot detect that --format became --output-format; a card that records mytool export --format csv can.

Card rules

Use references/scenario-card-template.md for every card:

• Required fields per card: stable ID and name, user goal, category, preconditions, exact inputs, steps, expected result, observations to capture, and safe variations. The fields are required; the layout is not — small libraries can use the full template, and libraries beyond roughly twenty cards should use the condensed one-card-per-bullet-block form shown in the template so files stay readable.
• Tailor every card to the target app. No placeholder paths, commands, controls, or expected behavior may remain.
• Group cards into numbered, topic-oriented files in dependency order. Give each file a one-line core question it answers (for example "Is the CLI robust to real terminal usage and misuse?").
• Create the library README from references/library-readme-template.md, recording safety constraints, the tailored execution contract, run order, files, pass shapes, and the coverage checklist.
• For libraries beyond roughly twenty cards, add a scenario index to the README (ID, file, name) so replays and reports can reference cards without re-reading every file.
• Add a scope table stating what the library deliberately does not cover because another surface already covers it (unit tests, static audits, component tests). The library targets behavior only detectable by running the app as a user; duplicating other test surfaces dilutes every pass.

Required coverage

Include at least one card per category, or record a specific not-applicable rationale in the library README:

1. First run or initial empty state
2. Primary happy-path workflow
3. Primary workflow with invalid input
4. Save or persistence behavior
5. Delete, remove, cancel, or undo behavior
6. Settings, preferences, or configuration
7. Surface sweep — every top-level command, screen, or route touched at least once
8. Close and relaunch behavior
9. Interrupted or stopped workflow
10. File or data import/export
11. Error recovery
12. Edge or boundary input

Invalid input means safe-but-wrong: letters in number fields, empty required fields, wrong file types, boundary values, malformed dates, oversized text. Never malicious payloads.

This list is a floor, not a ceiling. Extend it with categories the target app's domain demands — model or provider switching, permission and approval boundaries, concurrency and load, headless or server surfaces, migration from a prior release, multi-window behavior — and add the extensions to the library README's coverage checklist so replays inherit them.

Amendments

When a replay confirms app drift — the interface really changed — amend the library in a deliberate Part 1 pass: update the affected cards, bump the library version in the README with a one-line note of what changed and why. Old run reports stay untouched so history remains interpretable.

Part 2 — Replay the library ("the check")

One replay mechanism serves three checks. Which one a run performs is determined by what changed since the last run — and only one thing may change per comparison:

• New model stress test — a new model or model version drives the repo; same harness, same app commit.
• New tooling stress test — the agent harness, CLI, or toolchain changed; same model, same app commit.
• App release check — the app shipped a feature or dependency bump; same runner as the last known-good run.

To compare several models head-to-head, run the same pass once per model against the same app commit and library version, then read the cross-runner comparison matrix.

Run header

Every run report begins by identifying exactly who ran it and against what: runner model name and version, agent harness and version, date, target repo commit, library version, pass shape, and environment. Without this header, comparison across runs is guesswork.

When the target app itself invokes a model or agent (an AI tool, chatbot, agent harness), "the model" exists in two distinct roles and the header must record both separately:

• Runner — the model and harness driving the playtest from outside.
• App-side configuration — the provider and model the app is configured to use internally.

A new model variant can be appraised in either role. Changing the app-side model while keeping the same runner tests the app's model integration; changing the runner while pinning the app-side model tests whether the new model can still drive the interface. Never vary both in one comparison run.

Execution rules

• Read the library README first. Follow its documented order, dependencies, and safety limits. Default to the full pass; narrow only on explicit request and list every excluded card and why.
• Run cards verbatim. Type the exact commands, flags, and inputs the card records. Do not modernize, correct, or substitute an equivalent invocation. A card that only passes after a silent correction is a Fail with drift evidence, not a Pass — this rule is the mechanism that detects changed arguments, flags, and commands.
• The only sanctioned deviations are those the library's own safety rails force (a live operator instance, a port in use, credentials that must not be touched). Record every such deviation in the run report's Deviations section with its reason; a deviation that isn't recorded is a silent correction.
• First-run gates (onboarding wizards, setup dialogs, license screens) are part of the primary happy path. Complete them as a user would, with run-marker data, and record them; never bypass one through a backdoor and then report the workflow behind it as tested.
• Run CLI checks the way a script would (non-TTY, output captured). When behavior may differ on an interactive terminal — detaching, prompts, colors — note which mode you observed; a TTY/non-TTY behavioral divergence is itself a finding.
• Record a status per card: Pass / Fail / Partial / Blocked / Not applicable / Not executed.
• Evidence discipline: label every claim Confirmed (observed by running the app) or Suspicion (inferred from code or docs). Capture exact inputs, visible error text, logs, and observed state. Never claim a card was executed if it was only inferred. If the app could not be launched, say so and report the blocker instead of fabricating results.

Drift triage

Classify every non-Pass result:

• app_drift — the interface or behavior actually changed. Evidence: the recorded invocation fails identically under direct manual verification (help text, docs, a second attempt) or under a previously qualified runner.
• runner_drift — the interface is intact but the runner failed the card: invented flags, skipped steps, ignored preconditions, misread output. Evidence: manual verification or a previously qualified runner passes the same card unchanged.
• environment_drift — a missing dependency, credential, port, or service. Status is Blocked, not Fail.
• unresolved — the evidence to distinguish is unavailable; say so.

One failing run alone cannot tell app drift from runner drift. Confirm with a direct manual check of the recorded interface, or a re-run of that card with a previously qualified runner, before assigning blame to either side.

For targets that embed a model: behavior differences traced to a changed app-side model are app-side configuration drift — record them as app_drift with the configuration named, never as runner drift. Cards for such apps should pin deterministic fixtures (fixed prompts, mock providers, expected markers in output) so that model nondeterminism inside the app does not read as interface drift.

Comparison

When prior reports exist in runs/, build a card-by-run status matrix and call out every transition: Pass→Fail regressions, Fail→Pass recoveries, and newly Blocked cards. Summarize drift by class so the reader can see at a glance whether the app moved, the runner moved, or the environment broke.

Two comparison axes, read from the same matrix:

• Over time (same runner, successive runs): shows what a new app release or tooling change broke.
• Across runners (same app commit and library version, one column per model): shows which cards every model passes, which cards only stronger models pass, and which model to trust for this repo. Cards that only some runners pass are runner-sensitivity findings worth naming in the report — they mark the workflows where model choice actually matters.

Qualification gate

A runner (a specific model plus harness) is qualified for a repository when both hold:

1. It completed a full pass on the current library version with every card Pass or explained — Not applicable per the README, or Blocked with a named environment cause.
2. At least one other distinct runner has completed the same pass on the same library version.

Until both hold, the runner is unqualified: do not rely on it alone to produce final deliverables from this repository. Record the verdict in the run report and update the qualified-runners table in the library README.

Safety rails

• Confirm the target repository is trusted before running its code. If provenance is unknown, stop and get explicit human approval before launching anything.
• Check for a live instance first. Before launching anything, scan for a running instance of the target app on this machine (listening ports, processes, the default data home). A live instance is operator state: never stop, restart, reconfigure, or pair with it, and never rebuild artifacts it is running from — a rebuild swaps files under a live process. Run the pass against a disposable home and a different port, invoking prebuilt artifacts directly, and record the resulting step deviations in the run report.
• Run in a sandbox or disposable test environment with a disposable data home. Prefer the lowest-risk run mode.
• Scope every cleanup command (kills, deletions) to the disposable environment by its unique path or marker — never by a pattern broad enough to match the operator's instance.
• No real personal data. No real credentials unless explicitly provided for a test environment; otherwise mark credential-dependent cards Blocked.
• No exploitation payloads, credential attacks, destructive filesystem operations, or irreversible external changes.
• Clean up only resources created by this run. If ownership or reversibility of any data is uncertain, leave it in place, label it as test data where possible, and report the manual cleanup candidate.

Severity scale

• Critical — crash, data corruption, lost work, blocked primary workflow, or irreversible destructive action without warning.
• High — a major workflow fails, saved data is wrong, relaunch breaks state, or recovery requires technical help.
• Medium — a secondary workflow fails, messaging is unclear, settings do not persist, or behavior is inconsistent.
• Low — minor usability issue, confusing label, visual glitch.
• Note — observation or product question, not clearly a bug.

Do not

• Substitute an ad hoc scenario list when a library exists.
• Edit, regenerate, or append to cards during a replay.
• Silently correct a stale command and mark the card Pass.
• Claim a card was executed when it was only inferred.
• Blame the model, or the app, without triage evidence.
• Write run results into standing cards.
• Rewrite or delete old run reports.
• Use real personal data or real credentials in tests.

Output contract

Part 1 delivers: the library files created or amended, a summary of the interface inventory, coverage against the twelve categories with any not-applicable rationale, and whether the new library was run in the same session.

Part 2 delivers: a run report written to docs/test_scenarios/runs/ from references/run-report-template.md, plus a final answer stating the runner identity, pass shape, per-card statuses, drift classifications with evidence, the comparison against the prior run, the qualification verdict, and the recommended next pass. If launch was blocked, deliver the report with the exact blocker and no fabricated results.