/adversarial-review
Run an isolated, cross-model, mandate-to-find-problems review of either a change diff (post-implementation) or a KBD planning artifact (pre-implementation).
This is the mini's Node.js port. Every script under scripts/adversarial-review/
is a .mjs entry point over Node's built-in fetch — no python3, no curl,
no jq. The logic lives in lib/review/.
Progress Signals (MANDATORY)
Before building the review packet, emit:
Starting adversarial-review — <mode> <target>
After the normalized findings are written, emit:
Completed adversarial-review — <verdict> (<critical>/<warning>/<suggestion>)
Use the target and counts from the packet/findings files. Never guess them.
This is a different job from the existing gates:
- A deterministic checklist catches what was thought of in advance. Adversarial review hunts for what was not.
sycophancy-correctioncorrects existing text for excess agreeableness. Adversarial review generates a finding set from scratch — and then has its own report screened by sycophancy-correction (see Anti-Theater Gate).- Pre-spec gates (interrogation, coverage scoring) operate pre-spec. This skill operates on concrete artifacts and diffs.
Isolation contract
The judge is a fresh-context API call to an OpenAI-compatible
/v1/chat/completions gateway. It
receives only a review packet — diff or artifact, acceptance criteria or
goals, file tree, constraints — and never the producing session's chat
history. Isolation is structural, not honor-system: the session reads only
the normalized findings JSON.
The producer never grades itself. The packet records producer_model;
dispatch falls back from the judge role to the critic role when they would
match, and emits a JUDGE_MODEL_COLLISION warning when no alternative differs
(see references/isolation-and-routing.md). A packet with
producer_model: unknown makes that comparison pass trivially, so it is
recorded as cross_model_check: unverified-producer-unknown rather than
passed off as a clean cross-model review.
Modes
--mode diff — post-implementation (kbd-execute QA gate)
Reviews a completed change's diff against its acceptance criteria. Runs after a deterministic checklist gate passes (cheap check first, expensive judgment second) and before archive.
Packet contents: change diff, acceptance criteria (tasks.md / OpenSpec spec
/ verification.md), repo file tree, blocking constraints from
.kbd-orchestrator/constraints.md, producer_model.
--mode artifact — pre-implementation (assess / analyze / plan)
Vets a stage artifact before the stage hands off, so downstream stages run against reviewed inputs:
| Stage | Artifact(s) | Mandate focus |
|---|---|---|
assess |
assessment.md |
missed gaps; claims unsupported by the codebase |
analyze |
analysis.md, library-candidates.json |
build-vs-adopt blind spots; uninspected candidates; stale landscape |
plan |
plan.md |
wrong ordering; missing dependencies; untestable or ambiguous change criteria |
research |
report.md, <slug>.provenance.md, plan.md of a deep-research package |
invented or orphan citations; unanswered sub-questions; contradictions presented as settled; a frontmatter verification_status the sidecar does not support |
Packet contents: the artifact(s), phase goals.md, prior-stage handoff
summaries, constraints, producer_model.
The research target is the one artifact target that is not a KBD stage: it
takes --package <dir> instead of --phase, its goals are the run's query,
parameters, and sub-questions (read from checkpoint.json and plan.md), the
three files travel as separate fields (research_report,
research_provenance, research_plan) so the per-field cap applies to each,
review_focus names the failure classes, and packet.truncation is always
recorded. A package missing any of the three files is refused with exit 2.
node scripts/adversarial-review/build-review-packet.mjs --mode artifact --target research --package ~/.prometheus/research/<package_id> --out review/packet.json
node scripts/adversarial-review/dispatch-judge.mjs --mode artifact --packet review/packet.json --out review/findings.json
--mode skill / --mode agent — generated artifacts (creation gate)
Reviews something a generator just produced, so a skill or agent is judged
by a model that did not write it. Unlike the two modes above, --target is a
filesystem path and --phase is optional — a creator often runs outside any
KBD phase, and requiring one would make the gate unreachable where generation
actually happens.
| Mode | Target | Packet contents |
|---|---|---|
skill |
generated skill dir | SKILL.md, parsed frontmatter, script inventory, cross-reference map, validator output, original intent |
agent |
generated Cargo workspace | agent.toml, system_prompt.md, workspace members with per-crate purpose, mcp_servers, cargo check result, original intent |
--mode decision — an idea, before committing to it
Reviews a decision someone is about to make, not code. --target is a
single file, not a directory.
The mandate's core instruction is that the judge must not score novelty. Si, Hashimoto & Yang (2025) had 43 experts spend 100+ hours each executing randomly-assigned ideas: LLM ideas rated more novel before execution, then dropped on every metric after it, and the ranking flipped. A pre-execution novelty rating is evidence pointing the wrong way. The judge rates whether the reasoning survives contact with reality.
The packet parses the decision into decision / assumptions / falsifier and
records missing_fields. A decision stating no falsifier cannot be wrong about
anything, which is itself the defect — the packet surfaces that structurally,
without spending a judge call. It also carries prior_decisions (via pk search), so a decision that contradicts an earlier one is visible to the judge.
node scripts/adversarial-review/build-review-packet.mjs --mode decision --target decision.md --intent intent.md --out packet.json
node scripts/adversarial-review/dispatch-judge.mjs --mode decision --packet packet.json --out findings.json
Decision-mode findings additionally require confidence (0–100),
what_would_change_this, and a non-empty disconfirming array — see
Output contract. These are the automation-bias
countermeasures: a review that cannot say what would change its mind is
manufacturing certainty.
Ordering matters more than the analysis. For personal or hard-to-reverse
decisions, run commit-before-reveal.mjs record first. Showing someone the
analysis and then asking what they think produces agreement, not judgement —
confidence in AI predicts whether users scrutinise it at all. The gate exits 2
and writes no analysis until a judgement is on record.
Decisions and their outcomes persist via decision-log.mjs (record /
outcome / revisit), so a later decision can be checked against what actually
happened.
node scripts/adversarial-review/build-review-packet.mjs --mode skill --target dist/my-skill --intent spec.md --out packet.json
node scripts/adversarial-review/build-review-packet.mjs --mode agent --target ./my-agent --intent spec.md --out packet.json
Both are manifest-level: they record what each file is and does, never its body. A generated workspace is several crates of Rust that would not fit a judge's context and would bury the signal if it did. The contract is enforced, not merely intended — a packet whose descriptive fields contain shell-function or Rust syntax is refused with exit 2 and no packet is written.
Truncation is always recorded. Every field is capped
(PACKET_FIELD_CAP_BYTES, default 40000). A clipped field carries an inline
[TRUNCATED …] marker, and packet.truncation reports the cap and per-field
byte counts. The block is present even when nothing was cut, so "nothing was
dropped" is distinguishable from "this packet predates truncation recording" —
otherwise a judge could return PASS on material it never received.
--intent supplies what the artifact was asked to be. Without it the judge can
only assess internal consistency, never whether the result answers the request;
the packet warns when it is missing.
Workflow
1. preflight scripts/adversarial-review/preflight-models.mjs [MODEL_ROUTING] class=small
2. packet scripts/adversarial-review/build-review-packet.mjs [MODEL_ROUTING] class=small
3. judge scripts/adversarial-review/dispatch-judge.mjs [MODEL_ROUTING] class=frontier
4. gate scripts/adversarial-review/check-findings-sycophancy.mjs (anti-theater screen)
5. surface findings JSON → caller gate semantics
Concretely:
# 0. Load the gateway credential and declare the producer.
# KBD_PRODUCER_MODEL is REQUIRED for the guarantee this skill exists to make.
# The judge's collision check compares candidate != producer, so an unknown
# producer makes it pass trivially.
#
# Do NOT write ${KBD_PRODUCER_MODEL:-claude-opus-5}. A default does not fix
# the problem, it hides it: the check would then compare the judge against
# a guess, pass, and record verified-distinct for a comparison that never
# happened. Export the real value, or let the guard refuse.
set -a; . ~/.prometheus/kbd/secrets.env 2>/dev/null || true; set +a
export KBD_PRODUCER_MODEL="claude-opus-5" # ← the model running THIS session
# 1. Preflight (cached 24h at .kbd-orchestrator/model-preflight.json)
# Reports the gateway, the model per role, and WHICH config layer supplied it.
node scripts/adversarial-review/preflight-models.mjs
# 2. Build the packet
node scripts/adversarial-review/build-review-packet.mjs \
--mode diff --phase "$PHASE" --target "$CHANGE_ID" \
--out ".kbd-orchestrator/phases/$PHASE/review/$CHANGE_ID/packet.json"
# 3. Dispatch the judge
node scripts/adversarial-review/dispatch-judge.mjs \
--mode diff \
--packet ".kbd-orchestrator/phases/$PHASE/review/$CHANGE_ID/packet.json" \
--out ".kbd-orchestrator/phases/$PHASE/review/$CHANGE_ID/findings.json"
# 4. Anti-theater gate (exit 2 = rejected: re-dispatch once with feedback)
node scripts/adversarial-review/check-findings-sycophancy.mjs \
--findings ".kbd-orchestrator/phases/$PHASE/review/$CHANGE_ID/findings.json" \
--counter-key "adv-review-$CHANGE_ID"
Artifact mode replaces --target "$CHANGE_ID" with --target assess|analyze|plan.
Dispatch fallback chain
Warn, never silently degrade, never block the pipeline:
- REST gateway — an OpenAI-compatible
POST /v1/chat/completionsat the resolved gateway (openai-proxy on:8181, orliter-llm apion:4000). Full isolation, true cross-model.dispatch-judge.mjsexit 0, and the findings recordisolation_mode: rest-gateway:<url>pluscross_model_check. - Harness-native fresh-context subagent — when no gateway is reachable
(
dispatch-judge.mjsexit 3), the calling session dispatches a subagent (Agent tool / equivalent) whose prompt is exactly: the mode's mandate file- the packet JSON. Nothing else. Findings are logged with
"isolation_mode": "harness-native"— a weaker guarantee (same model family), stated, not hidden.
- the packet JSON. Nothing else. Findings are logged with
- Pending review (exit 4) — no judge available at all. Write a local
pending_reviewreceipt for the cumulative Git diff. Development may continue; final local certification requires a completed independent review receipt or an SSH-signed waiver.
Output contract
findings.json (schema: skills/adversarial-review/assets/schemas/findings.schema.json):
{
"mode": "diff",
"verdict": "BLOCK",
"judge_model": "openai/gpt-4o",
"isolation_mode": "rest-gateway:http://localhost:8181/v1",
"producer_model": "claude-opus-5",
"cross_model_check": "verified-distinct",
"findings": [
{
"severity": "CRITICAL",
"file": "src/auth/session.rs",
"line": 142,
"claim": "Token expiry is checked before refresh, allowing a replay window",
"evidence": "diff hunk @@ -138,+142 removes the pre-refresh revocation check required by AC-3",
"suggested_fix": "Re-check revocation after refresh, before issuing the new token"
}
]
}
Severity semantics:
| Severity | Diff mode | Artifact mode |
|---|---|---|
CRITICAL |
record a KBD blocker and set certification blocked; fix, then re-run the deterministic checklist and adversarial-review |
revise artifact and re-vet (max 2 rounds, then accept with an "Unresolved review findings" section appended so the next stage sees them) |
WARNING |
logged in the change's review dir; proceed to archive | appended to the stage handoff summary |
SUGGESTION |
informational | informational |
verdict is BLOCK iff at least one CRITICAL finding exists.
Anti-theater gate
The judge's findings report is itself screened through sycophancy-correction
(scripts/adversarial-review/check-findings-sycophancy.mjs, which spawns the
sycophancy-correction binary over a stdio MCP JSON-RPC session): a report
scoring ≥ 0.4 or matching high/critical sycophancy patterns —
e.g. zero findings on a large multi-file diff wrapped in hedged praise — is
rejected, and the judge is re-dispatched once with the rejection feedback
appended to its mandate (dispatch-judge.mjs --feedback <file>). A soft cap
then accepts the next report with a logged warning, mirroring the retry-loop gate.
When the sycophancy binary is absent the gate degrades gracefully (exit 0,
warning) — it never blocks the chain.
The rejection cap is yours to set
The cap defaults to 2 and is overridable via PROMETHEUS_ADV_REJECT_CAP:
PROMETHEUS_ADV_REJECT_CAP=4 node scripts/adversarial-review/check-findings-sycophancy.mjs --findings f.json
| Value | Effect |
|---|---|
| unset | cap 2 (default) |
| 1–5 | honoured |
| above 5 | exit 1 — refused, never clamped |
| 0, negative, non-numeric | exit 1 — refused, never silently defaulted |
A value above the ceiling is an error rather than being clamped down, because a silently-lowered cap would leave you believing a bound was in force that was not.
Every run records the cap in the findings artifact, so a stored review is auditable after the fact:
"sycophancy_screen": { "reject_cap": 4, "cap_overridden": true, "cap_default": 2 }
This cap bounds how many times an evasive judge report is sent back. The creators' 2-round retry cap — how many times an artifact is re-reviewed after CRITICAL findings — is a separate bound owned by
review-retry-loop.mjsand is unaffected by this variable.
Skip rules
- No file-count or documentation-only heuristic exists in any mode.
- Diff mode reviews the cumulative Git diff since the last accepted local review receipt, so a sequence of small commits cannot evade review.
--skip-qaand--skip-adversarial-revieware independent. Either recordspending_review; neither satisfies final local certification without an SSH-signed waiver.
Multi-model preflight
scripts/adversarial-review/preflight-models.mjs runs at KBD method start
(kbd-assess step 0) and lazily before any dispatch without a fresh cache. It:
Detects provider keys from the canonical env vars.
Resolves each role (
judge/critic/generator) against~/.prometheus/kbd/models.toml[roles], then reports which config layer supplied it. Two files own this, and neither is a script:~/.prometheus/kbd/models.toml— role → model NAME (KBD owns)~/.config/liter-llm/liter-llm-proxy.toml— NAME → provider + base_url +${KEY}(liter-llm owns)
Repair or extend both by hand, or with
/liter-llm-bridge configurewhen that skill is available. It merges and never clobbers.Verifies ≥ 2 distinct dispatchable models exist (so judge ≠ producer is always possible). Exactly 1 →
status: degraded. It also reportsconfig_defectswith named defects when the liter-llm config exists but cannot serve a request — missing[general] master_key(401 on everything) or a localhostbase_urlwith no[security] outbound_policy(deny_privateblocks loopback).When no keys are found: ask the user which providers to configure and instruct them to export the key. This skill never collects, stores, or writes API keys — config files hold aliases only; keys stay in the environment.
Cache: .kbd-orchestrator/model-preflight.json — re-run on --force,
config change, or age > 24 h.
Mobile portability classification (classify-mobile-execution.mjs)
Not part of the review pipeline above — a standalone tool that assigns every
script-bearing skill in skills/ a mobile execution verdict (E0/E1/E2/R),
derived from what its scripts actually invoke, not hand-typed:
node scripts/adversarial-review/classify-mobile-execution.mjs --out .kbd-orchestrator/phases/mobile-skill-portability/mobile-classification.json
node scripts/adversarial-review/classify-mobile-execution.mjs --check
Escalation — Party Mode is not built here
Multi-persona debate is deliberately out of scope. A single isolated adversarial reviewer captures most of the value, while fixed-round multi-agent debate can launder confidence into consensus — the exact failure mode sycophancy-correction exists to fight. Reserve multi-persona escalation for pre-implementation, hard-to-reverse decisions.
References
references/output-contract.md— findings schema and gate semanticsreferences/isolation-and-routing.md— fresh-context judging rationale, model resolution, collision handling, fallback chain, preflight contractreferences/model-configuration.md— how the judge finds a model that is not the producer, and how to point it at any provider you have