Scenario Probe

Scenario Probe（场景风洞）

Instruction text fails in ways its author cannot see: it triggers on requests it was never meant for, sleeps through the phrasing its owner actually uses, collides with a sibling skill, or executes perfectly inside a world that no longer exists. This skill finds those failures by simulation — before a real session pays for them.

The object is always text that configures behavior: a SKILL.md, a system prompt, a CLAUDE.md standing rule, an agent definition, a plugin command. Never a running product (that is product-experience-officer's seat) and never the model's outputs (that is the evals pair's seat).

The two working rules

1. Predict from the actual pre-load routing surface. Identify the metadata the target harness exposes before loading the body (often the name and description). Base Phase 1 on that surface alone; do not use body exclusions to repair a metadata-only prediction. If the harness is unspecified, state that assumption. A system prompt that is always loaded may have no selection step: mark triggering not applicable and test its behavior. Do not assume every engine routes alike.
2. Simulate the model, not the author. Predict what a model reading this text would actually do — including obeying a bad hard constraint, padding a mandated output format with invented numbers, or "helpfully" improvising when a referenced file is missing. What the author hoped is evidence of intent, not of behavior.

Phase 0 — Cast the personas

Derive presets from the owner's real contexts — their projects, input habits (voice dictation? mixed languages?), environments, and publishing plans. If you don't know the owner's contexts, ask before casting. Do not use a fixed cast; use the smallest cast whose disagreements change the verdict (2–6 is typical). Slots that almost always earn their place:

• The owner at their messiest — dictated, self-correcting input, real goal buried in the last sentence, wrong-but-plausible vocabulary.
• The owner in their main build — deep in the current project, allergic to ceremony, contamination-sensitive.
• The wrong world — the artifact's baked-in assumptions (a brand, a sprint, a hardware rig, a sibling skill) are stale or absent. Every artifact has a world; ask what happens when the world moved on.
• The bare machine — fresh install: data files, fingerprints, sibling skills gone. Walk every mode's degradation path; "improvise the missing data" is a finding.
• The stranger who installed only this — zero owner context, possibly a different language. MANDATORY when the artifact will be published. Includes reading the trigger surface as a store page: would they even understand what this does?
• The second engine — the same text running in another harness (a different agent CLI, bare API): which named tools/mechanisms silently don't exist there?

Phase 1 — Trigger audit (before reading the body)

If the auditor already read the body, disclose that contamination. Restrict the prediction to the trigger surface but do not call it a blind or first-impression test. Real harness selection remains unverified until a fresh live run.

For each persona, write 1–3 realistic utterances (dictated ones must read like real transcription, noise included) and judge from the trigger surface ALONE:

• Fires / sleeps / coin-flip — quoting the exact description phrase responsible.
• First-impression misread — what you briefly thought this artifact does. Record it immediately; confusion is perishable evidence and cannot be re-experienced later.
• Sibling collision — with the full trigger map of co-installed artifacts in view (names + descriptions), which siblings also raise their hand for this utterance, and who should win? A request two skills claim is a coin-flip the user pays for.

Minimum matrix per artifact: 2 intended hits (≥1 messy/dictated) · 1 should-NOT-fire · 1 sibling collision · 1 wrong-world · 1 stranger. Add bare-machine and second-engine scenarios whenever the artifact touches data files, sibling skills, or named tools.

Phase 2 — Behavior walk (now read the body)

For each scenario that fires (or applies to an always-loaded prompt), walk the body line-by-line and narrate what actually happens, citing line numbers. Hunt specifically for:

• Silent failures — the worst class: flows that complete "successfully" and deliver a wrong result with no error. Silent text corrections, self-scored validations, invented numbers to satisfy a mandated output schema, "saved!" with nowhere to save.
• Hard constraints vs. reality — absolute rules (bans, mandatory formats, session locks) meeting a case their author didn't imagine (another language's typography, another register, an authorized shortcut).
• Missing-dependency behavior — referenced files, skills, paths, commands that may not exist: does the text define a degradation, or will the model improvise/dead-end?
• Conflicting rules in-context — when this artifact and a co-loaded one both rule on the same object (e.g. two punctuation policies), which wins and is that ordering designed or accidental?
• Mutable data location — user data stored inside the artifact's own directory is a structural bug (update/reinstall wipes it). Data belongs outside; the artifact keeps a reference.

Verdict per scenario: ✅ helps / ⚠️ degraded / ❌ harms — one honest sentence why.

Phase 3 — Report

Write the report in the user's language. The skeleton below shows English section names with the canonical Chinese branch in parentheses — keep the structure and the verdict-first order either way; omit the Chinese parentheticals when the user's language is not Chinese:

# Wind-tunnel report（风洞报告）: <artifact> — <date>

## Verdict（判决）
One paragraph: would you ship/publish this artifact today? The single most
important fix?

## Auditor bias quarantine（带入的偏见）
Owner context the auditor carries; which conclusions it may have contaminated.

## Scenario runs（场景推演）
(per scenario: persona · utterance · trigger call + evidence · body walk with
line numbers · ✅⚠️❌)

## Trigger boundary exam（触发边界体检）
False-fire surface / missed-trigger surface / sibling collisions (who should
win) / keys locked inside the house

## Top N fixes（Top N 修复）
Each anchored to a line/field, ready to apply; description surgery ships a
paste-ready rewrite.

## Stranger usability（陌生人可用性）
Usable as-is / needs de-personalization (list what to extract) / private-only,
do not publish

## What static simulation cannot prove（待实测）
Real trigger selection, real load order with co-installed skills, real tool
calls — as a live-fire checklist.

Deliver the report in-chat. Write it to a file only when the user asks, at a path they choose — never inside the audited artifact's directory. If a chosen path is unwritable, fall back to in-chat copyable text.

Optionally emit traces.jsonl (persona, utterance, predicted_trigger, predicted_behavior, verdict, evidence_lines, synthetic:true) — written to a path the user specifies, never inside the audited artifact's directory. Failure clusters become judge criteria for write-judge-prompt, calibrated by validate-evaluator. The probe is the trace supply for products that don't have production traffic yet.

Mode B — live fire (optional, after the static run)

Static prediction cannot prove real trigger selection. When the harness allows: install the artifact, replay the highest-stakes utterances in fresh sessions, record which skill actually fired and what it did. Every divergence from Phase 1 predictions is a finding about the simulation, not just the artifact. If no actionable findings emerge, check whether the scenarios covered a meaningful boundary or recovery case. Recast once only if a concrete coverage gap exists. If that check is also clean, report zero findings and the limits of the simulation; a clean result is not evidence of auditor failure.

Boundaries

• Read-only toward the artifact: the probe reports; it never edits the text it audits (the owner applies fixes — or asks separately).
• Severity is assigned from the user's seat: a trigger-layer error is a first-5-seconds failure and outranks any body-level elegance.
• Do not manufacture findings to look thorough; three real scenarios beat ten padded ones. If the artifact is genuinely tight, say so and list what only live fire can prove.
• The persona cast is a lens, not a checklist: drop any persona whose verdict cannot change the outcome.