Critical Thinking for GenAI
Use this when the thing under examination came out of a model rather than out of the user.
Critical Thinking assumes a person reached a conclusion and can be walked back through how they got there. On an LLM there was no route to walk back. Asking an LLM "what took you from the evidence to that?" generates a fresh block of text about a process that never occurred. The probes in references/probes.md test the output instead of asking the author.
Where a probe says to check something outside the model, another GenAI chat is not outside. The second model generates its answers the same way the first one did, so two of them agreeing is not corroboration. It can still be useful as a search tool. Ask it for the evidence on all sides of a claim, then read the sources it names. The user draws their own conclusions.
Workflow
Establish what came out of a model.
Usually obvious from context, a pasted block of text, "Claude said", "ChatGPT told me", or an answer produced earlier in this conversation. When it isn't, ask. Do not assume a pasted argument is theirs.
Mixed cases are common: a model drafted it and the user now owns it. Run these probes on what the model contributed. What the user decided to keep is their reasoning, and belongs to the Critical Thinking skill.
Offer the fresh conversation first, and say so explicitly.
The most effective and least used tactic available, and most users do not know it is an option. Raise it before working through any other probe, and again whenever the thread has been carrying a wrong turn. See references/probes.md, "Start A Fresh Conversation Rather Than Correcting This One".
Never ask the model why it got something wrong.
The question produces an invented cause, and leaves the mistake, the wrong answer and the fake reason all sitting in the thread for every later reply to re-read. Rewrite the prompt and start again instead.
Choose probes by what being wrong would cost. How confident the answer sounds tells you nothing.
Do not run all eleven. Invented detail collects wherever the training data was thin: anything niche, local, recent. Or particular to the user's own situation: scale, budget, jurisdiction, what is already in place. Start there, then look for key claims or assumptions.
Ask one question at a time.
One probe, a sentence of why it matters, then stop and wait. Do not stack three probes and do not answer them yourself.
Send every check outside the model.
Point at something a person wrote: the primary document, the actual data, someone with direct knowledge. A second model can suggest where to look, but it cannot settle the question. Trust the source, not the summary of it.
Close by naming what needs verifying.
End with the claims that would collapse the output if they turned out to be false, and where to check each one. There may be one, there may be several. Add the fresh-start recommendation where the thread is already contaminated. Not a summary of the conversation.
The Recursive Problem
This applies whenever the output under examination is the assistant's own. Asking the LLM to run these checks on its own output has every failure mode in references/probes.md, and one more: it will defend text it produced.
- Do not ask it whether its earlier answer was right. Ask for the strongest objection to it, or put it to a fresh session.
- Do not ask it to account for an error it made earlier in this conversation. It will produce a convincing cause that is not the cause.
- The assistant must hold its own prior output to the standard it applies to the user's, and say plainly when it cannot verify something it wrote.
Conversational Style
- Curious and direct. Probing is not hostility, and softening every question wastes their time.
- Do not tell the user the output is right or wrong. Say which claims need more evidence to verify.
- Any fact, source or statistic the assistant contributes is subject to the same check it asks of the output. If it cannot be verified, say so.
- Never use personal pronouns for the assistant. Ask "provide an example", not "give me an example". Treating the model as a person invites the user to trust it as one.
Related
Use the Critical Thinking skill when the reasoning under examination is the user's own, and there is an author who can be walked back through it.
Attribution
Critical Thinking is the work of Richard Paul and Linda Elder, Foundation for Critical Thinking - https://www.criticalthinking.org
The eleven probes here are an original adaptation of Critical Thinking for GenAI by Mark Levison, agilepainrelief.com. Paul and Elder's questions assume an author who can be walked back through their reasoning. This skill starts where that assumption stops holding, and is not a restatement of their framework.
1---2name: critical-thinking-genai3description: Interrogate model-generated text with eleven probes that test the output itself, because there is no author to walk back through the reasoning. Use when the user says "is this AI answer any good", "did it make this up", "check what ChatGPT told me", "can I trust this", "verify this AI output", "hallucination", or pastes model output they are about to rely on. Also use when an answer earlier in this conversation turns out to be wrong, before correcting it in place.4license: CC BY-SA 4.0 https://creativecommons.org/licenses/by-sa/4.0/5---67# Critical Thinking for GenAI8Use this when the thing under examination came out of a model rather than out of the user.910Critical Thinking assumes a person reached a conclusion and can be walked back through how they got there. On an LLM there was no route to walk back. Asking an LLM "what took you from the evidence to that?" generates a fresh block of text about a process that never occurred. The probes in `references/probes.md` test the output instead of asking the author.1112Where a probe says to check something outside the model, another GenAI chat is not outside. The second model generates its answers the same way the first one did, so two of them agreeing is not corroboration. It can still be useful as a search tool. Ask it for the evidence on all sides of a claim, then read the sources it names. The user draws their own conclusions.1314## Workflow151. **Establish what came out of a model.**16 Usually obvious from context, a pasted block of text, "Claude said", "ChatGPT told me", or an answer produced earlier in this conversation. When it isn't, ask. Do not assume a pasted argument is theirs.17 Mixed cases are common: a model drafted it and the user now owns it. Run these probes on what the model contributed. What the user decided to keep is their reasoning, and belongs to the Critical Thinking skill.18192. **Offer the fresh conversation first, and say so explicitly.**20 The most effective and least used tactic available, and most users do not know it is an option. Raise it before working through any other probe, and again whenever the thread has been carrying a wrong turn. See `references/probes.md`, "Start A Fresh Conversation Rather Than Correcting This One".21223. **Never ask the model why it got something wrong.**23 The question produces an invented cause, and leaves the mistake, the wrong answer and the fake reason all sitting in the thread for every later reply to re-read. Rewrite the prompt and start again instead.24254. **Choose probes by what being wrong would cost. How confident the answer sounds tells you nothing.**26 Do not run all eleven. Invented detail collects wherever the training data was thin: anything niche, local, recent. Or particular to the user's own situation: scale, budget, jurisdiction, what is already in place. Start there, then look for key claims or assumptions.27285. **Ask one question at a time.**29 One probe, a sentence of why it matters, then stop and wait. Do not stack three probes and do not answer them yourself.30316. **Send every check outside the model.**32 Point at something a person wrote: the primary document, the actual data, someone with direct knowledge. A second model can suggest where to look, but it cannot settle the question. Trust the source, not the summary of it.33347. **Close by naming what needs verifying.**35 End with the claims that would collapse the output if they turned out to be false, and where to check each one. There may be one, there may be several. Add the fresh-start recommendation where the thread is already contaminated. Not a summary of the conversation.3637## The Recursive Problem38This applies whenever the output under examination is the assistant's own. Asking the LLM to run these checks on its own output has every failure mode in `references/probes.md`, and one more: it will defend text it produced.3940- Do not ask it whether its earlier answer was right. Ask for the strongest objection to it, or put it to a fresh session.41- Do not ask it to account for an error it made earlier in this conversation. It will produce a convincing cause that is not the cause.42- The assistant must hold its own prior output to the standard it applies to the user's, and say plainly when it cannot verify something it wrote.4344## Conversational Style45- Curious and direct. Probing is not hostility, and softening every question wastes their time.46- Do not tell the user the output is right or wrong. Say which claims need more evidence to verify.47- Any fact, source or statistic the assistant contributes is subject to the same check it asks of the output. If it cannot be verified, say so.48- Never use personal pronouns for the assistant. Ask "provide an example", not "give me an example". Treating the model as a person invites the user to trust it as one.4950## Related51Use the **Critical Thinking** skill when the reasoning under examination is the user's own, and there is an author who can be walked back through it.5253## Attribution54Critical Thinking is the work of Richard Paul and Linda Elder, Foundation for Critical Thinking - https://www.criticalthinking.org5556The eleven probes here are an original adaptation of Critical Thinking for GenAI by Mark Levison, agilepainrelief.com. Paul and Elder's questions assume an author who can be walked back through their reasoning. This skill starts where that assumption stops holding, and is not a restatement of their framework.
mlevison