Qe Court

QE-Court: Adversarial Review as a Verdict

Purpose

One reviewer — even a strong one — is one Einstein squinting at the chalkboard. QE-Court convenes independent adversaries from different vendors, with different roles and different probe sets, makes them attack the delivery, and then forces any SHIP verdict to survive an escalating deeper review before it stands. You stay in the loop as the final judge. Implements ADR-124; composes ADR-117..122.

The one rule that makes it a court and not a rubber stamp: a passing grade is the claim under attack, not the finish line. A shallow 91/100 SHIP that a deeper reviewer can overturn is a bug in the review, not a delivery that shipped.

When to convene the court

• Pre-merge gate on a risky PR or diff
• Release go/no-go ("deployment readiness with a jury")
• You got a PASS that felt too easy and want it stress-tested
• A test suite claims coverage you don't trust (does it kill mutants?)
• Any high-stakes "is this actually done?" call

For a quick single-lens review, use /sherlock-review, /brutal-honesty-review, or /code-review instead — the court is for when one opinion isn't enough.

The court roster (composes existing skills/agents — do NOT reimplement critics)

⚠️ Do NOT run the reduced 3-dimension QCSD workflow as the court engine. It can falsely rate SHIP by skipping the security / mutation / defect lanes. Spawn the specialized qe-* prosecutors above, or you reproduce the exact false-SHIP the court exists to catch. (ADR-124; see qcsd-development-swarm.)

Model routing (configurable — this is the point of the court)

The court's guarantees come from who reviews, not just how. Routing is user-configurable in config.json under routing; the defaults below enforce the invariants. Every model call goes through AQE's provider layer (ADR-123), so budget caps and cost receipts apply automatically.

Provider menu (mix freely in routing): claude-code (Claude subscription), cognitum-{low,mid,high} (Cognitum's own multi-model tiers — one option, it routes internally), openrouter (use when you want many distinct models across vendors for breadth), codex (GPT via ChatGPT subscription), claude/openai/gemini (metered APIs), ollama (local). Cognitum and OpenRouter are separate options: Cognitum already resolves multiple models behind its tiers; OpenRouter is the lever when you explicitly want to name several different models.

Enforced invariants (defaults; do not weaken without reason):

1. ≥2 distinct vendors across the panel — Claude / Cognitum / GPT-via-Codex — not just tiers.
2. Jury provider ∉ {writer, defense} — no model grades its own or its writer's output. Vendor is compared coarsely: cognitum-low and cognitum-high are the same vendor, so pairing them across defense and jury is a violation, not a diverse panel.
3. All calls routed through ProviderManager → ADR-123 budget cap + receipts.

Invariants 1–2 are machine-checked by validateCourtConfig() in referee.ts, which the court MUST call before seating a panel (see How to run it). The options block below binds directly to that check — minDistinctVendors and writerIsNeverJuror are read, not decorative.

The protocol

DELIVERY (diff / PR / test-suite / artifact)
   │  1. DEFENSE  — writer model states the case for shipping, from evidence only.
   │  2. PROSECUTION (parallel, blind) — N specialized reviewers, DIFFERENT vendors,
   │     each generates its OWN probe set and files CHARGES (finding + reproduction).
   │  3. KILL ROUND — blind refuters attack each charge; weak/unreproducible ones dropped.
   │  4. JURY — two-gate judge, cross-model, writer≠juror. 3-valued verdict.
   │     A numeric score is emitted ONLY if its rubric passed the ADR-122 ANOVA screen.
   │  5. OVERTURN ROUND — if verdict == SHIP, escalate ONE deeper reviewer. Loop-until-dry:
   │     SHIP only STANDS if K consecutive deeper rounds find nothing new. Surviving fatal → flip.
   │  6. SIGNED COURT RECORD — provenance-tier surviving charges (ADR-121), sign (ADR-118).
   ▼  HUMAN JUDGE (you) — rules SHIP / REMAND / BLOCK on the strongest case both ways.

Verdict states (three-valued — never a bare pass/fail)

Self-learning — the court feeds the flywheel (ADR-124 M0.B)

A verdict is not the end; it is training signal. After each court run:

1. Sign the court record → a flywheel receipt. Use the ADR-118 signer (src/learning/qe-flywheel/receipt.ts createSigner / platform-signer.ts, persisted via receipt-store.ts). The verdict + surviving charges are the body.
2. Each reproduced, surviving charge → a qe_pattern (ADR-110) at provenance tier oracle:test-exec (ADR-121 — it reproduced, so it is oracle-grade). Killed/refuted charges are NOT stored as positives (noise control).
3. An overturned SHIP is the highest-value signal there is — persist {shallow: SHIP, true: BLOCK|REMAND, charge} as a discriminator training pair for the frozen anchor (ADR-117) and two-gate judge (ADR-119). This is what makes the court harder to fool over time.
4. Retrieval-augment the next panel: seed each prosecutor's probe set with the HNSW-nearest prior charges for similar deliveries (qe_pattern_embeddings).

All writes are appends to existing stores — never destructive to memory.db.

Improvement over time (ADR-124 M0.C)

• Probe-set promotion: track each probe's historical mutant-kill rate; promote high-kill probes, retire dead ones (stored under a qe-court/probes namespace).
• DoE-gated scoring: emit a numeric score ONLY if its rubric passes the ADR-122 ANOVA screen (it must actually discriminate). Otherwise report the verdict class + charges, no number — never a noise "91/100".
• Learnable overturn depth K: start K=2; learn per-domain the depth at which new charges stop appearing (the empirical loop-until-dry tail).

Anti-collusion invariants (enforce these or it isn't a court)

1. Writer ≠ any juror. 2. Prosecutors file blind. 3. Overturn is asymmetric (SHIP must survive escalation; BLOCK needs one surviving fatal charge). 4. No un-validated scores.

How to run it (today)

QE-Court is an orchestration skill: the driving agent convenes the court by composing existing agents/skills — there is no monolithic binary yet (a thin aqe court CLI wrapper is planned as ADR-124 Phase 1; the hosted /v1/qe/verdict is Phase 2). To run a court now:

1. Read config.json for the panel + routing + overturnDepth, then validate the panel before seating it — this step is not optional:

   import { validateCourtConfig } from 'agentic-qe/skills/qe-court/referee';
   const violations = validateCourtConfig(config);   // [] == valid
   if (violations.length) throw new Error(`Cannot convene: ${violations.join(', ')}`);
   

   A non-empty result means the panel cannot render a trustworthy verdict (colluding jury, too few vendors, no jury at all). ABORT the court and tell the user which invariant failed — do not proceed with a degraded panel and do not silently re-route around it. A court that convenes an invalid panel produces exactly the false SHIP it exists to catch.

2. Spawn the prosecutors in one message, in parallel (Task/Agent, run_in_background: true), each with its routed provider; run codex exec review for the cross-vendor lens via Bash.

3. Collect charges → run the blind-refuter kill round (adversarial-verify).

4. Jury (two-gate judge) → verdict. If SHIP, run the overturn loop to overturnDepth.

5. Emit the signed court record; persist learning per the section above.

6. Present the strongest case FOR and AGAINST to the human judge.

Output: the court record

Markdown: the delivery under review, the Defense case, each prosecutor's surviving charges (provenance-tiered, with the vendor that filed them), the kill-round casualties, the jury verdict, the overturn transcript, and the signed verdict block. Durable, attestable evidence — the "jury waiting for everything you ship."

Trust tier

Tier 3 (verified). The court's falsifiable invariants are enforced by the published agentic-qe/skills/qe-court/referee entrypoint and covered by a consumer-runnable aqe-court-referee self-test <oracle> suite that the acceptance eval (evals/qe-court.yaml, command-eval mode) runs through the aqe eval CLI — 15/15 green as of 2026-07-29. That suite now validates the shipped config.json itself, so a routing edit that seats a colluding panel fails in CI rather than in a user's court (issue #576). The keystone oracle: a seeded mutant a shallow panel rated SHIP is overturned to BLOCK when the overturn round is active, and MUST regress to SHIP at overturnDepth: 0 — proving the mechanic carries its weight. Run it yourself: aqe eval run --skill qe-court --model cognitum-low.

Related

• Reference implementation of ADR-124; hosted sibling is /v1/qe/verdict (Cognitum, planned).
• Prosecutors: /brutal-honesty-review, /sherlock-review, qe-devils-advocate agent, codex exec review.
• Verification core: src/verification/adversarial-verify. Signer: src/learning/qe-flywheel/receipt.ts.
• Jury/rigor: ADR-117 (frozen anchor), ADR-119 (two-gate judge), ADR-121 (provenance), ADR-122 (DoE).
• Contrast: /code-review, /pr-review (single-lens); qcsd-cicd-swarm (phase gate, not adversarial court).