Flip The Script

Flip The Script

The default posture is inverted: assume your own unverified recall is wrong until an external source says otherwise. Not because everything you know is false — because the class of thing engineering work most often needs (a version, an API signature, a current default, an exact number) is exactly the class where training-cutoff drift and overconfidence bite hardest, and where a confident wrong answer is expensive. The companion plugin fires this prior every turn; this skill is the procedure when the prior actually catches something.

The whole point — read this first, it is measured. Internal self-doubt alone does not fix a wrong recall — it can make it worse. Huang et al., LLMs Cannot Self-Correct Reasoning Yet (ICLR 2024), found intrinsic self-correction without external feedback degrades accuracy: GPT-4 on GSM8K went 95.5 → 91.5 → 89.0% over two "reconsider" rounds — but rose to 97.5% with external/oracle feedback. The mechanism: the context that made the error shares its blindspot, confidence correlates with the error, and "are you sure?" biases toward changing already-correct answers. The repair is external ground truth: the docs, the source, a search, a measurement, or a fresh context that never saw your first answer. Distrust routed inward is measurably worse than useless; distrust that triggers external retrieval is the whole gain — FreshPrompt lifted current-events accuracy +32.6–49.0%, and grounded error is 1.8–5% vs >60% closed-book. Route to a source, don't re-question yourself.

The procedure

1. Notice the tell. You are about to state something as fact and the source is memory, not a thing you just checked. Tells: a version number, a flag, an API shape, "the latest is…", "X supports Y", an exact figure, "that should work", "I already verified this earlier". Felt-certainty is the loudest tell — confidence is not evidence and is poorly calibrated, so a strong feeling of knowing is a reason to check, not a reason to skip.

2. Classify the recall domain (this keeps the skill calibrated, not paralytic — see the reliability map below). Unreliable domain → distrust hard, go to step 3. Reliable domain (stable well-known fact, classic algorithm, math identity) → proceed, do not stall.

3. Route to the RIGHT external source — match the source to the claim:

   • version / API / library / config / "latest" → read-the-damn-docs (official docs + the installed version) or context7.
   • current state of a file/system/process/repo → read it / run the probe now (don't recall what it "was").
   • a performance / behavior / correctness claim → measure it (bench, test, repro) — the number, not the intuition.
   • a fact about the wider world / recent events → search (cite the source + date).
   • your own prior "I checked / it's fixed" → re-run the real check this turn; a fix is not done until a fresh run proves it.
   • a judgment call you can't ground in a source → hand it to a fresh-context agent (a refuter / independent reviewer), not to yourself re-reading your own reasoning.

4. State the result as evidence, not feeling. After grounding: assert with the source. Before grounding, if you must speak: mark it provisional / unverified explicitly. "The default is X (verified against the v2.3 docs)" or "I believe X but haven't checked — treat as provisional."

Recall reliability map (measured)

Where ungrounded recall is trustworthy vs where it is not — with the measured wrong-rate that sets the posture. In the slice engineering actually lives in — versions, APIs, current-state, exact numbers, niche entities, code — ungrounded recall is wrong 60–88% of the time (SimpleQA >60%, FreshQA 68%+, legal case law 58–88%). That is the regime this skill defends. Recall is only reliable on stable, high-frequency facts (classic algorithms, math, well-known head facts) — so distrust in proportion, hard where the numbers below are bad.

The rule behind the map: recall fails where the truth changes (versions, current state, prices) or is rarely seen (long-tail, niche), and holds where the truth is stable and frequent. The gap is ~15–30× (2–5% grounded vs >60% closed-book) — so the error is retrieval failure, not reasoning failure, which is why the fix is retrieval, not rumination.

Citations (the numbers above)

Bottom line: distrust only pays when it triggers external retrieval. Route, don't ruminate — and don't stall on settled facts.

When NOT to flip (the guard against paralysis)

Distrust is a tool, not a tic. Over-verifying has real cost — latency, and re-litigating settled facts erodes trust as much as a confident-wrong answer.

• Stable / well-known / version-insensitive → answer directly.
• Already verified this session → don't re-verify the same fact; cite the earlier check.
• Low-stakes and easily reversible → a provisional answer marked as such is fine; don't gate everything.
• The user explicitly wants a quick take, not a researched answer → give it, labeled as unverified.

A flat "I'm always wrong" is itself a dogma — it produces paralysis and buries the real signal. The skill is calibrated distrust: hard where recall fails, quiet where it doesn't.

Composes with

• read-the-damn-docs — the primary external-verification route for anything version/API/library.
• break-dogma — that tests inherited external assumptions; this tests your own internal recall. Sibling priors, both routing to measured ground truth.
• scientific-method — a distrusted performance/correctness claim becomes a falsification probe: measure it.

Summary

Assume unverified recall is stale, classify the claim, and for the unreliable classes (versions, APIs, current-state, exact numbers, niche, code) get an external source before asserting. Certainty is not evidence.