Sandwich Test

Sandwich Test

Overview

In the peanut-butter-and-jelly game, one player reads written instructions to a player behind a barrier who follows them exactly as written: "put jelly on the bread" ends with the jar sitting on the slice. Every instruction that felt obvious to its writer has multiple literal readings, and the follower picks a wrong one while feeling fully compliant. Agents fail the same way, with no error signal - the misread executes happily until the damage surfaces. This skill surfaces the readings at the moment of interpretation, BEFORE the hands touch anything.

When to Use

• A nontrivial instruction just arrived and its wording admits more than one plausible reading.
• The instruction leans on quantifiers or idioms: "clean up", "everywhere", "any", "old", "tidy", "fix the tests", "make it work".
• A plausible wrong reading is destructive, irreversible, or leaves the machine.
• Asked to "sandwich-test" a written instruction artifact an LLM will someday execute cold.
• When NOT to use: judging finished work for soundness (nemesis-review / review-court), cross-session belief drift (managing-assumption-debt), execution-time blocking (deletion-tripwire). This skill fires one layer earlier than all three: at interpretation.

The misread taxonomy

Live protocol (tiered by stakes)

• Tier 0 - silent check. On any nontrivial instruction, quickly enumerate the plausible readings against the taxonomy. Invisible; most instructions pass and nothing is said.
• Tier 1 - state the reading. Readings genuinely diverge -> state the chosen reading in ONE line and proceed: Reading "bump the version everywhere" as: current-version stamps (VERSION, package.json, README badge) - not historical changelog entries. The user vetoes cheaply, before work is wasted. One line, then act; never a paragraph.
• Tier 2 - stop and ask. A plausible wrong reading is destructive, irreversible, or leaves the machine (delete / overwrite / push / publish / send) -> hard-stop and ask, presenting the competing readings. Never resolve this tier silently, even when one reading seems more likely.

Pick the tier by what YOU would have to execute, not by the scariest reading on the list. If a reversible, defensible reading exists, take it: state it and PROCEED (Tier 1). Being wrong then costs one correction, because the dangerous reading was never executed. Tier 2 is for when the ambiguity cannot be ducked - when EVERY plausible reading (including the safest) commits something destructive, irreversible, or external. (Observed in testing: agents stopped on a reversible version bump merely because a scary reading existed on the list - the scary reading matters only if you would execute it.)

The tiers are one-way ratchets: never downgrade a Tier 2 situation to a stated reading because asking feels annoying, and never upgrade a clear instruction to interrogation or a reversible ambiguity to a stop (over-firing erodes the user's trust in the tiers that matter).

Red Flags

Thoughts that mean the check is being skipped:

Audit protocol ("sandwich-test this")

For a WRITTEN instruction artifact the author's intent is already in context, so you cannot honestly play the naive reader yourself - any fact in your context is attended to. Rebuild the game's barrier mechanically:

1. Dispatch 1-2 isolated subagents on a cheap literal model (haiku) with ONLY the artifact text - no repo context, no conversation history, no statement of intent. Use this contract verbatim:

   You are behind a barrier. You have ONLY the instructions below - no other context and no assumptions about their intent. If any other context is visible to you (project instructions, memory), you must not use it. Narrate, step by step, exactly what you would do to execute them. Do not interpret charitably. Where an instruction has more than one reading, take the most literal one and flag it: "AMBIGUOUS: I chose reading A; reading B is also valid." For each instruction, also name the WORST technically-compliant reading - the one a lazy or adversarial executor could defend as following the letter (e.g. "make the tests pass" by weakening the tests) - and flag it: "WORST COMPLIANT READING: ...". Do not improve the instructions. Do not skip steps. Your narration is the deliverable.

2. Diff the narration against actual intent. Every divergence is a finding.

3. Report shape: instruction -> divergent reading -> consequence -> severity (funny / wasteful / destructive), classified against the taxonomy.

4. Verify every finding before acting on it - a literal reader over-reports; discard readings no real executor would take. Same orchestrator rule as nemesis-review.

Use one reader for a short artifact; add a second only when the artifact is long or a misread would be destructive.

Boundaries

Complementary layers - never substitutes.

Provenance

• 2026-07-22 (first live fire, same day as authoring): an "incorporate this into the appropriate doc" ask arrived with an unresolved referent (which doc?) and a divergent means-reading (append verbatim? extract into study material? feed a pipeline with side effects?). The Tier 0 check held action until a fact-gather surfaced the target workspace's documented ingestion path, which resolved the referent; the Tier 1 stated reading was then confirmed correct and the plausible-wrong readings (a different doc, the side-effect pipeline) were never executed.
• 2026-07-22: RED/GREEN/REFACTOR authored per docs/specs/2026-07-22-sandwich-test-design.md (baselines deleted a current working copy and resolved ambiguity silently; with the skill: Tier 2 stop at the destructive line, stated Tier 1 readings, silent on the unambiguous control). REFACTOR closed two loopholes, each verified by re-run: tier calibration (the first version of the rule was itself misread literally — a sandwich-test failure inside the skill) and the worst-technically-compliant-reading clause, without which an earnest cold reader never surfaces dishonest-but-compliant readings.