Intuition Probe

Intuition probe

Use a model's prior as a cheap sample of familiar interface conventions. A divergent guess is a design candidate, not merely a model error. Prefer conforming the interface to a convergent familiar guess unless safety or a hard invariant forbids it.

This is adapted for Pi from Jeremy Theocharis's intuition-probe, itself based on Anselm Eickhoff's Jazz principle: “make the API whatever the LLM guesses.”

Guardrails

• Ordering is the experiment: freeze the blind prompt, then inspect reality, then launch probes.
• Never expose artifact paths, implementation details, exact identifiers, or the answer key to a probe.
• One sample produces candidates only. Call something a priority only when at least two independent samples converge.
• Launch probes through the sub-agents skill's runner. They are isolated pi -p processes with tools, skills, context files, prompt templates, and sessions disabled; extensions remain enabled for provider support.
• Never edit the artifact under test as part of this skill; report recommendations only.

Procedure

1. Ask for the artifact, rough outcome, optional read-set, and sample count (default 1). Do not inspect the artifact yet.

2. Rewrite the outcome without leaked method names, keys, flags, labels, paths, or implementation clues. Ask the user to confirm the sanitized goal.

3. Read references/blind-prompt.md. Fill its placeholders in a temporary prompt file. A read-set is allowed verbatim; fetch URL content and inline it rather than giving the probe a URL. Freeze this file now.

4. Only now inspect the real API signatures, schema/examples, CLI help/definitions, or UI routes/components. This is the answer key and stays in the orchestrator context.

5. Read the sub-agents skill, then run the probe wrapper from this skill directory:

   bash scripts/run-blind.sh /path/to/frozen-prompt.md N
   

   The wrapper reuses the generic sub-agent runner with a frozen repeated prompt, no tools, and probe-specific JSON validation. It uses openai-codex/gpt-5.6-luna by default and prints an output directory containing probe-*.json and logs. Read every result. If a probe emits invalid JSON, preserve it as a failed sample; do not quietly repair its design choices.

   Judge the batch unsatisfactory when it is malformed, empty, substantially hedged instead of committing to an interface, or fails to address the sanitized outcome. Do not reject it merely because its guess differs from reality—that divergence is the point. If unsatisfactory, retry the same frozen prompt and N with Terra:

   PI_INTUITION_MODEL=openai-codex/gpt-5.6-terra bash scripts/run-blind.sh /path/to/frozen-prompt.md N
   

   Keep both batches and report that fallback occurred. If Terra is also unsatisfactory, stop and report the failed probe rather than silently changing the prompt.

6. Compare every decision against reality using references/scoring.md. Fold duplicate/hedged decisions explicitly rather than dropping them.

7. Group equivalent guesses across samples. Convergence means the same semantic shape, not merely similar wording.

8. Report:

   • mode (cold or doc-informed), model, and N;
   • the candidate interface/spec the probes reached for;
   • divergences with bucket, confidence/convergence, familiar anchor, and conform-first recommendation;
   • matches briefly;
   • rejected conform moves and the concrete safety/invariant reason;
   • limitations: same-model samples are repeated draws, not independent human usability evidence.

For N=1, label the report candidate only — not a confirmed familiar default. Offer to save it outside the repository under test.