Manual Testing Playbook Creation
create-manual-testing-playbook is the manual-validation package workflow for the sk-doc family. It authors manual-testing-playbook/ packages for skills and systems that need reproducible operator-facing scenarios, evidence capture, release-readiness review, and realistic orchestration or multi-agent execution planning.
Core principle: keep shared rules in the root playbook, keep execution truth in per-feature files, and make every scenario deterministic enough that another operator can reproduce the verdict.
1. WHEN TO USE
Activation Triggers
Use this workflow when the request involves:
/create:manual-testing-playbook.- Creating
manual-testing-playbook/manual-testing-playbook.md. - Building one-file-per-feature manual test scenarios.
- Designing deterministic prompts, command sequences, expected signals, evidence, pass/fail criteria, and failure triage.
- Planning manual validation across multiple operators, agents, CLIs, MCP tools, or runtime surfaces.
- Converting an ad hoc release checklist into a reusable evidence-driven playbook package.
- Aligning a manual testing playbook with a feature catalog.
Keyword triggers: manual testing playbook, /create:manual-testing-playbook, testing playbook, playbook system, deterministic scenario, evidence collection, operator validation, multi-agent execution, release readiness, playbook conversation, multi-turn conversation script, playbook scenario.
Strong signals that a playbook is warranted:
- 5+ distinct features need manual validation.
- Release decisions depend on structured evidence.
- Realistic orchestration behavior matters.
- Multiple operators or agents will execute the same scenarios.
- Automated tests cover internals but not operator-visible behavior.
When NOT to Use
Use another sk-doc packet when:
- Test steps fit cleanly in a spec folder checklist.
- The feature is one-off or experimental.
- The system has only a few manually testable behaviors.
- Automated tests already cover the only meaningful acceptance criteria.
- The user asks for a feature catalog rather than executable validation scenarios. Use
create-feature-catalog. - The user asks for a README, skill, agent, command, benchmark package, flowchart, or changelog. Use
create-readme,create-skill,create-agent,create-command,create-benchmark,create-flowchart, orcreate-changelog. - The task audits, validates, scores, or optimizes an existing playbook without creating or extending the package. Use
create-quality-control.
2. SMART ROUTING
The sk-doc hub routes manual validation here: the request names a manual testing playbook, a deterministic scenario, evidence collection, operator validation or a playbook conversation, and the hub's sk-create-manual-testing-playbook mode signal selects this packet.
It creates a playbook only when reusable manual validation needs captured evidence, not when test steps fit a spec-folder checklist.
ROUTER.md scores three scopes: the root playbook, a per-feature scenario and a multi-turn conversation scenario. Each loads its own resources, and a request that names several loads their union.
references/README.md is the fallback route map, and authoring starts from assets/manual-testing-playbook-template.md for the root playbook, assets/manual-testing-playbook-snippet-template.md for each single-prompt scenario file, assets/manual-testing-playbook-conversation-snippet-template.md for each conversation scenario file and references/conversation-scenarios.md for the conversation script format.
Read ROUTER.md before selecting skill-local resources; open only the matched files.
3. OUTPUT PACKAGE CONTRACT
Canonical Package Shape
Author this layout:
manual-testing-playbook/
|-- manual-testing-playbook.md
|-- category-name/
| |-- feature-name.md
| `-- another-feature-name.md
`-- another-category/
`-- feature-name.md
Package invariants:
- The root file is always
manual-testing-playbook.md. - Per-feature files live in root-level category folders.
- Category directories use descriptive kebab-case names such as
category-name(no numeric prefix). - Per-feature files use stable kebab-case slugs such as
feature-name.md; no numeric file prefix. - Per-feature snippet order is defined by the root playbook listing order.
- Display order is owned by the root playbook index (
manual-testing-playbook.md), not the folder name. - Benchmark tier is owned by the per-feature file's optional
stage:frontmatter field (routingdefault, orholdout/negative), not by a filename token. - Every feature ID maps to exactly one per-feature file.
Two runtimes, where a system ships as both a skill and a hosted assistant:
- Write two scenario sets, never one system tested twice. Category folders carry a
prefix naming the runtime, and every per-feature file carries a matching
- Runtime:line inSOURCE METADATA. - Each set opens with one identity handover scenario, named as a precondition by every other scenario in that set. It passes only when the reply carries a verbatim string present in that runtime's own instructions and absent from the other's, plus the delivery contract only that runtime can honour. A reply that could have come from either runtime is a FAIL.
- A handover string is proved against each runtime's whole load surface, not against its top-level instruction file alone. A skill runtime loads that file and everything the file sends it to. A hosted runtime loads its own instructions and every document attached to it. A string absent from one file and present three directories away has not been proved.
- Read the string at run time where it can move. Grading on a literal version line fails the next time that version is bumped, which makes a version check wearing an identity check's clothes.
Contract boundary:
- The operator-scenario contract is the five-section, execution-evidence contract
defined here and checked by
scripts/validate-playbook-package.cjs. - The routing-gold contract is the compact typed-routing schema checked by
sk-create-skill/scripts/validate-playbook-topology.cjs. playbook-corpus-manifest.jsonis an explicit whole-tree override map consumed only by the operator validator. A listed routing-gold tree is excluded from operator checks; for every other file, a non-emptyexpected_workflow_modeplus at least oneexpected_leaf_resourcespair in frontmatter classifies that file as routing gold. Files without that signature are operator-scenario files. The topology gate does not read this manifest and keeps its existing boundary.
Do not create:
- A
snippets/subtree for canonical per-feature files. - Separate canonical
review_protocol.md. - Separate canonical
subagent_utilization_ledger.md. - Duplicated scenario truth across multiple sidecar files.
Root Playbook Responsibilities
manual-testing-playbook/manual-testing-playbook.md is the package directory and review surface. It owns:
- Frontmatter and H1 intro.
- Global overview and coverage note.
- Global preconditions.
- Global evidence requirements.
- Deterministic command notation rules.
- Integrated review protocol and release-readiness rules.
- Integrated orchestration and wave-planning guidance.
- Category sections with short per-feature summaries.
- Automated test cross-reference section.
- Feature catalog cross-reference index.
Root summaries should be concise but useful:
- Provide enough context to understand what each scenario covers.
- Preview operator intent and prompt shape.
- Do not duplicate the full execution matrix from the per-feature file.
Root-to-feature rule: the root document explains package-level policy; per-feature files carry scenario-specific execution truth.
Per-Feature File Responsibilities
Each per-feature file is the canonical scenario contract for one feature ID.
Required per-feature section order:
## 1. OVERVIEW## 2. SCENARIO CONTRACT## 3. TEST EXECUTION## 4. REFERENCESor## 4. SOURCE FILES## 5. SOURCE METADATA
Each per-feature file must include:
- Frontmatter with
title,description, and a 4-partversion. - Realistic user request when it clarifies user intent.
- Operator prompt or orchestrator prompt.
- Exact prompt in the scenario table when a table is used.
- Exact command sequence.
- Expected signals.
- Evidence requirements.
- Pass/fail criteria.
- Failure triage.
- Root playbook link.
- Feature catalog link when applicable.
4. RESULTS STORAGE CONTRACT
A playbook is a corpus. Running it produces evidence, and that evidence has a home: the same
benchmark/ tree that holds every other measurement of the skill. Without this rule a run leaves
nothing behind, and the next person has no way to know the playbook was ever executed.
Where A Run Lands
<skill>/
|-- manual-testing-playbook/ # the corpus, an input, never rewritten by a run
`-- benchmark/
`-- reports/
|-- README.md # the run index, one row per folder
`-- 2026-07-29--manual-testing-playbook--goal-hook/
|-- README.md
|-- skill-benchmark-report.json
|-- skill-benchmark-report.md
|-- results.csv
|-- failed-runs.md
|-- findings-and-recommendations.md
`-- source.md
Run folders are named <YYYY-MM-DD>--<subject>--<variant>, dated by execution. When a run is
feature-scoped — a hand-derived validation of one feature or scenario group rather than a full-corpus
harness sweep — name the <variant> for the feature (e.g. goal-hook) and record the model/executor
inside the report, so the folder stays legible across models. create-benchmark owns the grammar in full;
see its naming convention section for the field vocabulary and the one carve-out.
Manual scenario completion
A manual scenario is incomplete until its PASS, FAIL, or SKIP outcome, its reason and its evidence paths are
recorded in the skill's benchmark/reports/<dated-run-label>/ folder. The wrapper that once persisted them
was retired with the Lane C harness, so the record is written by hand until a replacement exists.
Rules
- The corpus is an input. A run never edits
manual-testing-playbook/, and gold that needs to change gets a corpus revision rather than a rewritten scenario. - Curated summaries and result tables live in the run folder. Raw transcripts and copied artifacts stay
in the spec packet that produced them, named in
source.md. - A run whose result changes gets a new folder. A prior run is never overwritten.
- Every file in the folder derives from the run record. A field the run did not capture reads as not recorded; it is never filled in from inference.
5. HOW IT WORKS - AUTHORING WORKFLOW
Follow this sequence:
- Confirm the target skill or system, package owner, feature set, and whether a feature catalog already exists.
- Decide whether a manual testing playbook is appropriate using the decision rule in this skill.
- Define category directories using a descriptive kebab-case slug such as
category-name. - Define stable feature IDs using a consistent
{PREFIX}-{NNN}pattern. - Create the root
manual-testing-playbook/directory. - Create
manual-testing-playbook/manual-testing-playbook.mdfromassets/manual-testing-playbook-template.md. - Create one per-feature file for each feature ID. Pick its style first with Choosing the Scenario Style. Start a single-prompt file from
assets/manual-testing-playbook-snippet-template.mdand a conversation file fromassets/manual-testing-playbook-conversation-snippet-template.md. - Write root package policy before writing scenario-specific exceptions.
- Write each per-feature prompt and execution truth before polishing root summary prose.
- Fill each scenario contract with the nine display fields:
- Feature ID, Feature Name, Scenario Objective, Exact Prompt, Exact Command Sequence, Expected Signals, Evidence, Pass/Fail Criteria and Failure Triage.
- The operator validator's eight unconditional checks cover Feature ID plus the seven execution/link fields.
- Realistic user request, table prompt and feature-catalog link are conditional checks.
- Write root category summaries and link every per-feature file.
- Add automated-test anchors when they exist.
- Add feature-catalog cross-references when a catalog exists.
- Explicitly note when a scenario has no dedicated catalog entry.
- Review destructive scenarios and isolate them in root preconditions and recovery guidance.
- Validate the root document with shared validators.
- Manually spot-check per-feature structure, prompt synchronization, feature ID counts, and local links.
- Report any remaining manual validation scope honestly.
Authoring sequence matters:
- Decide categories and IDs before writing summaries.
- Write root package policy before scenario exceptions.
- Write execution truth before root prose.
- Keep scenario-specific execution truth in per-feature files.
6. SCENARIO DESIGN RULES
Choosing the Scenario Style
Every scenario file uses one of two styles. Pick the style for each scenario before you write its execution truth. This section is the single home of that choice. The command workflow, the router and the references point here.
- Single prompt. The operator sends the Exact Prompt once, reads the answer and judges it against the Expected Signals. The operator collects the evidence by hand.
- Conversation. Section 3 carries one
json conversationblock. The shared runner plays each turn into a live runtime session. It answers the runtime's questions, restarts or compacts the session and runs a shell check after every turn. The Exact Prompt is the first turn'ssend, and the Exact Command Sequence is the runner command. The style changes how the scenario runs, not the file's shape.
Apply these questions in order and stop at the first yes:
- Does the verdict need a person's judgment, such as tone, document quality or whether an explanation is correct? Write a single prompt.
- Does the behavior need more than one exchange? That means the runtime asks a question back, an answer must change what happens next or state must survive a restart or a compaction. Write a conversation.
- Must the behavior hold on several runtimes, or does the scenario gate a release as a repeatable regression check? Write a conversation, provided a shell command can decide every turn.
- Otherwise write a single prompt.
Question 1 comes first because a conversation turn passes only when its check exits 0. When a multi-turn behavior also needs a judged verdict, script the turns a check can decide, and leave the judged part to the operator in the Pass/Fail Criteria, read from that turn's screen.txt.
A conversation also needs a runtime the runner drives (claude-code, opencode, codex, pi or devin), tmux, that runtime installed and logged in and quota to spend. One pass of the runner's own six-turn scenario on all five runtimes took about 11 minutes. The write sandbox needs macOS. Elsewhere the run records sandbox: unavailable and the subject runs unsandboxed. When any of these is missing, write a single prompt, or keep the conversation and record SKIP with the missing piece as its blocker.
A conversation catches failures that a single exchange never reaches. The runner's own live passes caught a start-up dialog that redrew before it settled (LAUNCH_TIMEOUT), a check that failed on one runtime only because of letter case, a spent usage limit (QUOTA_EXHAUSTED) and another process writing into the repository during the run (REPO_CHANGED). A single prompt costs no runner setup and works with any runtime, so keep it for routing checks, one command's output and anything a reader must judge.
The script is one fenced json conversation block under a ### Conversation Script heading in section 3. See conversation-scenarios.md for the block schema and turn rules. Play a script into a live runtime with the shared runner, from the repository root:
node .skilled/skills/sk-doc/sk-create-manual-testing-playbook/scripts/run-conversation-scenario.cjs \
--scenario <scenario.md> --runtime <id[,id...]> --scratch <dir outside the repository> --out <evidence dir>
It writes one PASS, FAIL or SKIP row per runtime to <evidence dir>/matrix.md, with the evidence for each beside it.
Determinism
Each scenario must be reproducible by another operator. Include exact prompts, exact command sequences, observable expected signals, captured evidence, and binary pass/fail criteria.
Execution status is limited to:
PASSFAILSKIPwith a specific sandbox blocker
Do not classify scenarios outside the PASS / FAIL / SKIP enum. A SKIP must name a specific sandbox or runtime blocker.
Prompt Quality
Prompts should be:
- Realistic, not bare command paraphrases.
- Deterministic enough to produce stable evidence.
- Explicit about what to capture.
- Explicit about the user-facing verdict or outcome.
- Compact enough for a 9-column table cell.
Weak prompt:
Test search
Acceptable prompt:
Use advisor_recommend for the flaky routing-threshold issue, capture the returned recommendation set, and return a concise pass/fail verdict with the main reason.
Natural-Human vs RCAF Voice
The canonical Prompt: field defaults to natural-human voice. Match how a real user would phrase the request to an AI in conversation.
Use the RCAF wrapper only when the actor is an AI orchestrator:
As a {ROLE}, {ACTION} against {TARGET}. Verify {EXPECTED_OUTCOME}. Return {OUTPUT_FORMAT}.
Use natural-human voice when:
- A human asks the AI directly in conversation.
- The scenario is a code review, commit/git workflow, bug fix, code question, or preference question.
- The scenario could plausibly originate from a Slack DM or terminal prompt.
Use RCAF when:
- The actor is an AI orchestrator delegating to another tool, AI, agent, CLI, or runtime.
- Role context determines safety refusal behavior.
- The validation contract treats the orchestrator as the operator.
When in doubt, prefer natural-human voice. The Real user request: field is always natural-human and serves as the voice reference baseline.
Prompt Synchronization Gate
These fields must agree:
- Structured prompt field in
SCENARIO CONTRACT. Exact Promptcolumn in the execution table.- Any root summary prompt text.
- A conversation chain's Turn 1 input, when the scenario has one, because that is the copy a runner sends.
Do not ship unsynchronized prompt fields. scripts/validate-playbook-package.cjs reports each copy that disagrees with the contract as a PROMPT_UNSYNCED warning. Quotes, backticks, emphasis and spacing are presentation and are ignored. Any difference in the words is reported.
7. VALIDATION AND RELEASE GATES
Shared Document Validation
Run shared validation on the root playbook before delivery from the repo root (replace <SKILL_PATH> with the target skill directory, e.g. .skilled/skills/system-spec-kit):
# New-content naming guard. The staging root must contain only the newly authored
# canonical manual-testing-playbook package, never an ancestor with shipped legacy roots.
python3 .skilled/skills/sk-doc/shared/scripts/check_no_hyphenated_catalog_content.py <new-content-staging-root>
python3 .skilled/skills/sk-doc/shared/scripts/validate_document.py <SKILL_PATH>/manual-testing-playbook/manual-testing-playbook.md --type reference
python3 .skilled/skills/sk-doc/shared/scripts/extract_structure.py <SKILL_PATH>/manual-testing-playbook/manual-testing-playbook.md
The staging scope is mandatory until shipped underscore roots are migrated. Do not run this guard against .skilled/skills or another ancestor containing legacy feature_catalog/ or manual_testing_playbook/ trees.
Also check:
- Broken local links, by grep/script or the repository markdown-link guard.
- Feature ID counts between the root index and per-feature files.
- Automated-test links, when referenced.
- Feature-catalog links, when referenced.
- The operator validator reports parser findings for a malformed
json conversationblock asCONVERSATION_SCRIPT_INVALID, naming the field path.
Operator Contract Validator
Run the operator-contract validator from the repository root:
node .skilled/skills/sk-doc/sk-create-manual-testing-playbook/scripts/validate-playbook-package.cjs \
--package .skilled/skills/<skill-id>/manual-testing-playbook
A fleet run scans every root, including packet-owned ones. discoverPackages descends into
<skill>/<packet>/manual-testing-playbook as well as <skill>/manual-testing-playbook, because a
packet keeps its playbook as a SIBLING of the hub's, not inside it. Enumerating only the first
level once left most of this corpus unscanned: a fleet run reported success while roots it had
never opened carried hundreds of violations. If you change discovery, re-check the scanned count
against find .skilled/skills -type d -name manual-testing-playbook | wc -l — a root that is not
scanned cannot fail, so absence looks exactly like success.
Make a clean package permanent. When a package reaches zero violations, remove it from
warnPackages in playbook-corpus-manifest.json in the same change, and add each of its roots to
playbook-failclosed-allowlist.txt. Graduating is what converts a one-time cleanup into a standing
guarantee: a warn-listed package reports its violations and ships anyway. The CI workflow
.github/workflows/playbook-operator-contract.yml runs the fleet validator and separately asserts
every allowlisted root is still discovered, so a root cannot quietly leave enforcement by being
renamed or moved. Fix a failing package rather than removing its line.
The command validates the operator-scenario contract, not the routing-gold contract. It walks every non-excluded scenario tree and checks, per feature:
- Five-section ordering.
title,descriptionand four-partversionfrontmatter.- Feature ID.
- Operator or orchestrator prompt.
- Exact command sequence.
- Expected signals.
- Evidence.
- Pass/fail criteria.
- Failure triage.
- Root-playbook link.
- Allowed verdicts.
- Filename and category shape.
- Unique IDs.
- Root-index bijection.
- Local links.
- Evergreen truth.
- Placeholder exclusion.
Conditional checks cover a realistic user request when user intent is explicitly being clarified, an exact prompt when a scenario table is present, and a feature-catalog link when catalog applicability is declared.
The validator derives scenario and category counts at run time. A root's hand-typed census is reported as a warning, including a mismatch, so documentation repair remains separate from enforcement. Existing measured packages are listed in the validator's staged warning set for the first fleet run; clean packages and new playbooks fail closed. Promotion removes a package from that warning set only after a clean run.
The validator also checks the root playbook for the result-persistence completion marker and the complete PASS / FAIL / SKIP
vocabulary as advisory warnings. Missing either item never creates a new fail-closed violation for an existing package.
Prompt synchronization is advisory in the same way: a prompt copy that disagrees with the scenario contract is a
PROMPT_UNSYNCED warning, never a violation.
Exit codes are direct: 0 means conforming or staged warning, 1 means a fail-closed contract violation, and 2
means a usage or boundary error. Strict mode is on by default; --no-strict is local triage only and must not be
used by CI.
The existing validate-playbook-topology.cjs remains the routing-gold consumer and is intentionally unchanged.
It continues reading its current playbook paths without consulting the additive corpus manifest.
8. RULES
✅ ALWAYS
- Use
manual-testing-playbook.mdas the root file name. - Put per-feature files in root-level category folders named with the bare descriptive slug.
- Keep one canonical per-feature file for each feature ID.
- Use exact prompts, exact command sequences, observable expected signals, captured evidence, and binary pass/fail criteria.
- Keep shared review, orchestration, and release-readiness rules in the root playbook.
- Link scenarios back to the matching feature catalog entry when a catalog exists.
- Explicitly document exceptions when no matching catalog entry exists.
- Mark destructive scenarios and include safe recovery expectations.
- Validate the root playbook and manually review per-feature files before delivery.
⛔ NEVER
- Create a
snippets/subtree for canonical per-feature files. - Create separate canonical
review_protocol.mdorsubagent_utilization_ledger.mdfiles. - Duplicate full execution truth in the root summary.
- Renumber published feature IDs just to remove gaps.
- Use packet-history or spec-phase references as scenario identity.
- Ship unsynchronized prompt fields.
- Add packet-local
graph-metadata.json. - Add numeric prefixes to category folder names or per-feature filenames; the root playbook index owns display order and the per-feature
stage:field owns benchmark tier. - Never use underscores in category folder or per-feature filename path segments.
⚠️ ESCALATE IF
- The target feature set, category boundary, or package owner is unclear.
- A scenario would require destructive actions without a safe recovery path.
- The user expects external credentials, live production access, or privileged tool execution.
- Feature catalog links are required but no stable catalog exists.
- Validation fails after structural fixes.
- The requested package shape conflicts with existing published playbooks.
9. SUCCESS CRITERIA
A create-manual-testing-playbook run is done when:
- The package uses the canonical shape: root
manual-testing-playbook.mdplus kebab-case category folders of per-feature files, with nosnippets/subtree and no separatereview_protocol.mdorsubagent_utilization_ledger.md. - Every feature ID maps to exactly one per-feature file, and no packet-local
graph-metadata.jsonwas added. - Every scenario is deterministic: exact prompt, exact command sequence, observable expected signals, captured evidence, and binary
PASS/FAIL/SKIPverdicts. - Prompt fields are synchronized across the scenario contract, the execution table, and any root summary.
- Destructive scenarios are marked and isolated with safe recovery expectations.
- The root playbook passes shared
validate_document.py, per-feature files are manually spot-checked, and any remaining manual scope is documented honestly.
10. RESOURCES FOR DEEP DETAIL & REFERENCES
The core executable workflow lives in this SKILL.md. Use these only for overflow detail, exhaustive examples, or template text:
references/README.md- reference map routing to the overflow detail below.references/prompt-voice.md- natural-human vs RCAF decision table and voice guidelines.references/conversation-scenarios.md- multi-turn conversation script schema, outcomes and evidence.scripts/run-conversation-scenario.cjs- the shared conversation runner that plays ajson conversationscript into a live runtime.references/common-pitfalls.md- recurring package defects and correct fixes.references/examples.md- shipped reference playbooks and scaffold templates.assets/manual-testing-playbook-template.md- root playbook scaffold.assets/manual-testing-playbook-snippet-template.md- per-feature single-prompt scenario scaffold.assets/manual-testing-playbook-conversation-snippet-template.md- per-feature conversation scenario scaffold.../shared/references/core-standards.md- shared markdown structure rules.../shared/references/validation.md- shared validation and DQI workflow.../sk-create-frontmatter/references/frontmatter-versioning.md- 4-part version expectations.../shared/references/evergreen-packet-id-rule.md- evergreen current-state wording.
The source assets keep the filenames manual-testing-playbook-template.md, manual-testing-playbook-snippet-template.md and manual-testing-playbook-conversation-snippet-template.md until their separate source-file migration. Those filenames are authoring inputs, not emitted package names.