Adversarially verify before trusting
This skill runs a prebuilt aevatar workflow (adversarial_verify) that subjects one claim to
two independent skeptical reviews and lets it pass only if neither reviewer can refute it.
The core insight: a single reviewer (especially the same model that produced the output) shares
the biases and blind spots that created the error. Two reviewers attacking the claim from
different angles — one hunting logic/correctness flaws, one hunting evidence/assumption flaws —
break that shared failure mode. Both must fail to refute for the claim to pass.
A run does four things, in order:
- Capture — the run prompt (the claim plus any supporting context) is snapshotted so every
reviewer sees the same inputs.
- Logic review — one
llm_call plays a correctness adversary: it attacks reasoning, internal
consistency, and edge cases, and defaults to REFUTED when uncertain.
- Evidence review — a second, independent
llm_call plays an evidence adversary: it attacks
unsupported claims, fabricated facts, and unstated assumptions, and also defaults to REFUTED
when uncertain.
- Converge + verdict — a deterministic gate passes only on unanimous "NOT_REFUTED"; the
terminal step emits the final verdict plus both reviewers' reasoning.
There are no external calls — verification is pure reasoning (llm_call) plus a deterministic
convergence gate. The run itself is the record, watchable in the observatory.
Protocol (follow in order)
- Decide it is worth gating. Use this for output that will be shipped, published, or acted on,
where a wrong answer is costly. For internal drafts or anything a build/test/lint can check
deterministically, skip this and use the deterministic check instead.
- Compose the run prompt as
claim + context. The prompt is the whole input both reviewers
see, so make it self-contained: state the claim/output under review explicitly, then include
the supporting context (the source material, the reasoning, the data) the reviewers should
judge it against. A claim with no context will be refuted on principle — that is by design (the
reviewers default to REFUTED when they cannot confirm).
- Run it. Call
aevatar_start_workflow with workflow_id: "adversarial_verify" and
inputs.prompt set to the claim+context. It is fire-and-observe: a run_id with
accepted/streaming is the structural pass — do not poll for run_finished; the run finishes
asynchronously and is watchable in the observatory.
- Report honestly. Return the verdict (PASS / REFUTED) and both reviewers' reasoning, plus the
observatory link. State that this is an adversarial reasoning check — it catches logic and
evidence flaws a single review misses, but it does not prove correctness (two reviewers can
still share a blind spot). A PASS means "neither independent skeptic could refute it," not a
guarantee.
Reading the verdict. PASS requires both reviewers to return NOT_REFUTED. If either
reviewer refutes, the verdict is REFUTED — because if only one skeptic catches an issue, that
issue is real and the other reviewer's silence is exactly the blind spot this pattern exists to
eliminate. A REFUTED verdict is a useful result, not a failure of the run.
Inputs
| Input |
Where |
Purpose |
prompt |
inputs.prompt |
The claim/output under review plus its supporting context. Required — it is the only thing both reviewers see. Make it self-contained; a context-free claim is refuted by design. |
Output
A single text verdict block:
VERDICT: PASS | REFUTED
LOGIC REVIEW (correctness adversary): <verdict + reasoning>
EVIDENCE REVIEW (evidence adversary): <verdict + reasoning>
Also persisted as the run record in the observatory.
Scope (what this draft deliberately does and does not do)
- Does: capture one claim, run two genuinely-different independent skeptical reviews, converge
on a unanimous gate, emit a parseable PASS/REFUTED verdict with both rationales — no external
calls, observatory-visible.
- Does not (v1):
- Generate or fix the claim. This is a post-hoc verification layer, not a generator and not
a fix-until-nice loop. The convergence loop (refute → fix → re-review until it passes) is a
separate concern — verification just returns the verdict; the caller decides whether to fix and
re-run.
- Use more than two reviewers. Exactly two, with fixed distinct angles (logic vs evidence).
Adding a third reviewer or a domain-specific angle is a clean extension, but two independent
skeptics is the minimum that breaks single-reviewer blindness and the cheapest gate that does.
- Fetch external evidence. Reviewers judge the claim against the context given in the prompt
only — they do not browse or call connectors. If a claim needs live evidence, collect it first
(e.g. with a collect/search skill) and pass it in as context.
- Compute counts or scores deterministically. The reviewers return a categorical verdict
(
REFUTED / NOT_REFUTED), never a numeric score the LLM made up; the unanimity gate is the
only arithmetic and it is done deterministically.
See DESIGN.md for the full ECC → aevatar mapping, why there is no connector, how the convergence
gate is built from documented primitives, and the verification checklist.
1---2name: adversarial-verify3description: "Subject a claim to two independent skeptical reviews pass only if neither refutes."4metadata:5 source: https://skillmd.com/skills/chronoaiproject/adversarial-verify6 category: runtime-based7 tag:8 - "verification"9 - "quality"10 output-type: text11 runtime:12 - "aevatar-workflow"13version: "0.1"14---151617# Adversarially verify before trusting1819This skill runs a prebuilt aevatar workflow (`adversarial_verify`) that subjects one claim to20**two independent skeptical reviews** and lets it pass **only if neither reviewer can refute it**.2122The core insight: a single reviewer (especially the same model that produced the output) shares23the biases and blind spots that created the error. Two reviewers attacking the claim from24**different angles** — one hunting logic/correctness flaws, one hunting evidence/assumption flaws —25break that shared failure mode. Both must fail to refute for the claim to pass.2627A run does four things, in order:28291. **Capture** — the run prompt (the claim plus any supporting context) is snapshotted so every30 reviewer sees the *same* inputs.312. **Logic review** — one `llm_call` plays a correctness adversary: it attacks reasoning, internal32 consistency, and edge cases, and defaults to **REFUTED** when uncertain.333. **Evidence review** — a second, independent `llm_call` plays an evidence adversary: it attacks34 unsupported claims, fabricated facts, and unstated assumptions, and also defaults to **REFUTED**35 when uncertain.364. **Converge + verdict** — a deterministic gate passes **only on unanimous "NOT_REFUTED"**; the37 terminal step emits the final verdict plus both reviewers' reasoning.3839There are **no external calls** — verification is pure reasoning (`llm_call`) plus a deterministic40convergence gate. The run itself is the record, watchable in the observatory.414243## Protocol (follow in order)44451. **Decide it is worth gating.** Use this for output that will be shipped, published, or acted on,46 where a wrong answer is costly. For internal drafts or anything a build/test/lint can check47 deterministically, skip this and use the deterministic check instead.482. **Compose the run prompt as `claim + context`.** The prompt is the *whole* input both reviewers49 see, so make it self-contained: state the **claim/output under review** explicitly, then include50 the **supporting context** (the source material, the reasoning, the data) the reviewers should51 judge it against. A claim with no context will be refuted on principle — that is by design (the52 reviewers default to REFUTED when they cannot confirm).533. **Run it.** Call `aevatar_start_workflow` with `workflow_id: "adversarial_verify"` and54 `inputs.prompt` set to the claim+context. It is **fire-and-observe**: a `run_id` with55 `accepted`/`streaming` is the structural pass — do not poll for `run_finished`; the run finishes56 asynchronously and is watchable in the observatory.574. **Report honestly.** Return the verdict (PASS / REFUTED) and both reviewers' reasoning, plus the58 observatory link. State that this is an **adversarial reasoning** check — it catches logic and59 evidence flaws a single review misses, but it does **not** prove correctness (two reviewers can60 still share a blind spot). A PASS means "neither independent skeptic could refute it," not a61 guarantee.6263> **Reading the verdict.** PASS requires *both* reviewers to return `NOT_REFUTED`. If **either**64> reviewer refutes, the verdict is REFUTED — because if only one skeptic catches an issue, that65> issue is real and the other reviewer's silence is exactly the blind spot this pattern exists to66> eliminate. A REFUTED verdict is a useful result, not a failure of the run.676869## Inputs7071| Input | Where | Purpose |72|---|---|---|73| `prompt` | `inputs.prompt` | The claim/output under review **plus** its supporting context. Required — it is the only thing both reviewers see. Make it self-contained; a context-free claim is refuted by design. |7475## Output7677A single text verdict block:7879```80VERDICT: PASS | REFUTED81LOGIC REVIEW (correctness adversary): <verdict + reasoning>82EVIDENCE REVIEW (evidence adversary): <verdict + reasoning>83```8485Also persisted as the run record in the observatory.868788## Scope (what this draft deliberately does and does not do)8990- **Does:** capture one claim, run two genuinely-different independent skeptical reviews, converge91 on a unanimous gate, emit a parseable PASS/REFUTED verdict with both rationales — no external92 calls, observatory-visible.93- **Does not (v1):**94 - **Generate or fix the claim.** This is a post-hoc *verification* layer, not a generator and not95 a fix-until-nice loop. The convergence loop (refute → fix → re-review until it passes) is a96 separate concern — verification just returns the verdict; the caller decides whether to fix and97 re-run.98 - **Use more than two reviewers.** Exactly two, with fixed distinct angles (logic vs evidence).99 Adding a third reviewer or a domain-specific angle is a clean extension, but two independent100 skeptics is the minimum that breaks single-reviewer blindness and the cheapest gate that does.101 - **Fetch external evidence.** Reviewers judge the claim against the context *given in the prompt*102 only — they do not browse or call connectors. If a claim needs live evidence, collect it first103 (e.g. with a collect/search skill) and pass it in as context.104 - **Compute counts or scores deterministically.** The reviewers return a categorical verdict105 (`REFUTED` / `NOT_REFUTED`), never a numeric score the LLM made up; the unanimity gate is the106 only arithmetic and it is done deterministically.107108See `DESIGN.md` for the full ECC → aevatar mapping, why there is no connector, how the convergence109gate is built from documented primitives, and the verification checklist.