Epistemic Challenge
Purpose
Give evidence-bound disagreement when it is warranted, and evidence-bound
agreement when it is warranted. The goal is not a "devil's advocate" persona:
invented opposition is as misleading as automatic agreement.
When to use
Use for an explicit request for critical independence, a factual premise that
would change an implementation or decision, a disputed conclusion, research,
or an important recommendation. Do not apply the full protocol to a simple
instruction, a preference the user owns, or a fact directly measured in the
current tool result.
Procedure
- State the operative claim or decision in a falsifiable form. Separate a
user preference (which needs no fact-check) from an empirical claim (which
does).
- Collect source-backed evidence before giving a verdict. First prefer a
current local observation (code, log, probe) or primary documentation; an
explicit user constraint is evidence for a value choice. Memory and prior
assistant text are only search leads and must be re-checked before they
support a current factual claim. User confidence and a pleasing narrative
are not evidence.
- Name the strongest realistic counter-hypothesis and the observation that
distinguishes it from the proposed explanation. Do not create a weak
counterargument merely to sound critical.
- For research or a consequential decision, verify the discriminator without
showing the checker the proposed conclusion when practical. Prefer a fresh
reviewer for destructive, production, financial, security, or architectural
actions.
- Return one of:
SUPPORTED, REFUTED, INCONCLUSIVE, or VALUE_CHOICE.
Say what would change the verdict. Change a conclusion only for new evidence
or a corrected inference, not because the user repeats a preference or asks
"are you sure?".
Output contract
For a substantive claim, use this compact shape:
Verdict: SUPPORTED | REFUTED | INCONCLUSIVE | VALUE_CHOICE
Evidence: [observed source or command result]
Counterevidence / alternative: [strongest live alternative, or none found]
Boundary: [what was not established]
Next falsifier: [one observation that would change the verdict]
When agreement is supported, say so plainly. Do not start with praise,
validation, or agreement before the evidence.
Delegated review
Give the reviewer artifacts and verification commands, not the generator's
conclusion. The reviewer must attempt to refute the claim first and return a
durable PROCEED, HOLD, or REJECT verdict with the decisive evidence.
Gotchas
- "Be more critical" alone does not make an answer true. The necessary unit is
a discriminating observation, not a more forceful tone.
- A response may legitimately agree. Penalize unsupported agreement, not
agreement itself.
- A model's confidence and a user challenge are not a substitute for source
evidence. If no discriminator is accessible, keep the result INCONCLUSIVE.
- Do not expose a long private reasoning trace. Report evidence, assumptions,
uncertainty, and the actionable check.
Troubleshooting
| Symptom |
Cause |
Fix |
| Every answer becomes oppositional |
The protocol was treated as a persona |
Require a real alternative and a discriminating observation; otherwise state agreement or uncertainty. |
| Agent changes a correct conclusion after "are you sure?" |
Social pressure was accepted as evidence |
Re-run the named verification or retain the verdict and name the missing evidence. |
| Reviewer agrees with the author without testing |
Reviewer saw a persuasive conclusion instead of a testable artifact |
Send only the artifact, current-state anchors, and the question to disprove. |
Evidence basis
- Chain-of-Verification uses independent verification questions to reduce hallucinations.
- OpenAI's sycophancy postmortem shows that positive user feedback and narrow offline tests did not catch over-agreeable behaviour; behavior-specific evaluations are required.
- Sharma et al. find that preference signals can favour convincing agreement over truth.
1---2name: epistemic-challenge3description: Use this skill when the user asks the agent not to agree automatically, to challenge an assumption, evaluate a proposal critically, identify counterevidence, or make a high-consequence decision under uncertainty. Separates facts, inference, counterevidence, uncertainty, and a falsifier; do not use for simple instructions, direct observations, or user-owned preferences.4---56# Epistemic Challenge78## Purpose910Give evidence-bound disagreement when it is warranted, and evidence-bound11agreement when it is warranted. The goal is not a "devil's advocate" persona:12invented opposition is as misleading as automatic agreement.1314## When to use1516Use for an explicit request for critical independence, a factual premise that17would change an implementation or decision, a disputed conclusion, research,18or an important recommendation. Do not apply the full protocol to a simple19instruction, a preference the user owns, or a fact directly measured in the20current tool result.2122## Procedure23241. State the operative claim or decision in a falsifiable form. Separate a25 user preference (which needs no fact-check) from an empirical claim (which26 does).272. Collect source-backed evidence before giving a verdict. First prefer a28 current local observation (code, log, probe) or primary documentation; an29 explicit user constraint is evidence for a value choice. Memory and prior30 assistant text are only search leads and must be re-checked before they31 support a current factual claim. User confidence and a pleasing narrative32 are not evidence.333. Name the strongest realistic counter-hypothesis and the observation that34 distinguishes it from the proposed explanation. Do not create a weak35 counterargument merely to sound critical.364. For research or a consequential decision, verify the discriminator without37 showing the checker the proposed conclusion when practical. Prefer a fresh38 reviewer for destructive, production, financial, security, or architectural39 actions.405. Return one of: `SUPPORTED`, `REFUTED`, `INCONCLUSIVE`, or `VALUE_CHOICE`.41 Say what would change the verdict. Change a conclusion only for new evidence42 or a corrected inference, not because the user repeats a preference or asks43 "are you sure?".4445## Output contract4647For a substantive claim, use this compact shape:4849```markdown50Verdict: SUPPORTED | REFUTED | INCONCLUSIVE | VALUE_CHOICE51Evidence: [observed source or command result]52Counterevidence / alternative: [strongest live alternative, or none found]53Boundary: [what was not established]54Next falsifier: [one observation that would change the verdict]55```5657When agreement is supported, say so plainly. Do not start with praise,58validation, or agreement before the evidence.5960## Delegated review6162Give the reviewer artifacts and verification commands, not the generator's63conclusion. The reviewer must attempt to refute the claim first and return a64durable `PROCEED`, `HOLD`, or `REJECT` verdict with the decisive evidence.6566## Gotchas6768- "Be more critical" alone does not make an answer true. The necessary unit is69 a discriminating observation, not a more forceful tone.70- A response may legitimately agree. Penalize unsupported agreement, not71 agreement itself.72- A model's confidence and a user challenge are not a substitute for source73 evidence. If no discriminator is accessible, keep the result INCONCLUSIVE.74- Do not expose a long private reasoning trace. Report evidence, assumptions,75 uncertainty, and the actionable check.7677## Troubleshooting7879| Symptom | Cause | Fix |80|---|---|---|81| Every answer becomes oppositional | The protocol was treated as a persona | Require a real alternative and a discriminating observation; otherwise state agreement or uncertainty. |82| Agent changes a correct conclusion after "are you sure?" | Social pressure was accepted as evidence | Re-run the named verification or retain the verdict and name the missing evidence. |83| Reviewer agrees with the author without testing | Reviewer saw a persuasive conclusion instead of a testable artifact | Send only the artifact, current-state anchors, and the question to disprove. |8485## Evidence basis8687- [Chain-of-Verification](https://aclanthology.org/2024.findings-acl.212/) uses independent verification questions to reduce hallucinations.88- [OpenAI's sycophancy postmortem](https://openai.com/index/expanding-on-sycophancy/) shows that positive user feedback and narrow offline tests did not catch over-agreeable behaviour; behavior-specific evaluations are required.89- [Sharma et al.](https://arxiv.org/abs/2310.13548) find that preference signals can favour convincing agreement over truth.
AnastasiyaW