# Deep QA Ensemble V1

> Use when the user asks to review with ensemble judges, or explicitly requests the ensemble variant of deep-qa. Benchmark variant that uses three heterogeneous providers for severity judging instead of a single judge. Same triggers as deep-qa but with multi-provider ensemble scoring.

- Skill: `npow/deep-qa-ensemble-v1` (Agent Skill, multi-file: 6 files)
- Install (CLI): `npx skillmds@latest add npow/deep-qa-ensemble-v1`
- Raw SKILL.md: https://api.skillmd.com/api/skills/npow/deep-qa-ensemble-v1/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: npow (https://skillmd.com/u/npow)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/npow/deep-qa-ensemble-v1

---


# Deep QA Skill — Ensemble Variant v1

> **Benchmark variant.** This skill is structurally identical to [`deep-qa`](../deep-qa/SKILL.md) except for severity-judge batches in Phase 3 step 10 (pass-1 blind) and Phase 5.5.b (pass-2 informed). Both phases now spawn a **three-model judge panel per batch** — Sonnet 4.6 via `Task` tool + GPT-5.4 via `mcp__pal__chat` + Gemini 2.5 Pro via `mcp__pal__chat` — with explicit verdict aggregation (majority-vote severity, max confidence, fail-up tiebreaker, preserved per-model rationales). All other contracts — Phases 0, 1, 2, 3 steps 1–9, 4, 5, 5.5.a coherence, 5.6 rationalization audit, 6 — are unchanged from baseline. Ensemble logic lives below in the "Ensemble Judge Panel" section; diffs from baseline are marked with ▶ in the affected phases.

Systematically audit an existing artifact for defects using parallel critic agents across QA dimensions tailored to the artifact type. Unlike deep-design (which designs and iterates) or deep-research (which explores the web), deep-qa takes an artifact as-is and finds what's wrong with it.

**No spec drafting. No redesign. Find and report.**

## Execution Model

Shares deep-design's core execution contracts:

- **All data passed to agents via files, never inline.** Artifact, known-defects list, angle definitions — all written to disk before spawning.
- **State written before agent spawn, not after.** `spawn_time_iso` written before Agent tool call. Spawn failure records `spawn_failed` status.
- **Structured output is the contract; free-text is ignored.** Severity judges produce machine-parseable structured lines. Unparseable output → fail-safe critical.
- **No coordinator self-review of anything load-bearing.** Severity classification delegated to independent judge agents.
- **Termination labels are honest.** Seven defined labels map to all reachable termination paths — never "no defects remain." See Phase 5 for the complete vocabulary.
- **Hard stop is unconditional.** `hard_stop = max_rounds * 2` is set at initialization and checked at the start of every round before any user prompt. No extension can exceed it.

**Shared contracts:** this skill inherits the four execution-model contracts (files-not-inline, state-before-agent-spawn, structured-output, independence-invariant) from [`_shared/execution-model-contracts.md`](../_shared/execution-model-contracts.md). The items listed above are the skill-specific elaborations; the shared file is authoritative for the base contracts.

**Cross-finding coherence:** this skill applies the coherence-integrator pattern from [`_shared/cross-finding-coherence.md`](../_shared/cross-finding-coherence.md) at Phase 5.5.a-coherence — after draining pass-1 judges and BEFORE pass-2 informed judges. The integrator reads all deduped critic output files simultaneously and annotates each finding with cross-finding relationships (contradictions, emergent patterns, coverage gaps). These annotations are included in pass-2 judge input files so judges see the cross-finding context when confirming/upgrading/downgrading severity.

**Subagent watchdog:** every `run_in_background=true` spawn in this skill (severity judges, coordinator summaries, batched pass-2 judges) MUST be armed with a staleness monitor per [`_shared/subagent-watchdog.md`](../_shared/subagent-watchdog.md). Use Flavor A (Monitor tail per spawn) with thresholds `STALE=3 min`, `HUNG=10 min` for Haiku judges and summaries — these are short-running tasks and a 30-min quiet period is always pathological. `TaskOutput` status field is not evidence of progress; output-file mtime is. This contract adds `timed_out_heartbeat` to this skill's termination vocabulary (per-lane watchdog kill) and `stalled_watchdog` / `hung_killed` to per-lane state — see shared doc §"State schema additions" + §"Termination-label addition".

## Adversarial judging (3 of 4 mechanisms adopted)

See [`_shared/adversarial-judging.md`](../_shared/adversarial-judging.md) for the full pattern: blind severity protocol, mandatory author counter-response, rationalization auditor, falsifiability drop.

Current deep-qa adoption status:

| Mechanism | Adopted? | Location |
|---|---|---|
| Independent judges (baseline) | ✅ yes | Severity classification is delegated to independent Haiku batches in Phase 3 step 10. |
| Blind severity protocol (two-pass) | ✅ yes | Phase 3 step 10 strips critic-proposed severity before pass-1 judge spawn; Phase 5.5.b runs pass-2 informed judges that may confirm/upgrade/downgrade; calibration signal logged if confirm rate is 0% or 100%. |
| Mandatory author counter-response | ✅ yes | Critic template requires an `Author counter-response` field — if the critic cannot write a plausible defense, the defect is filed as a minor observation instead of a defect. |
| Rationalization auditor | ✅ yes | Phase 5.6 spawns an independent auditor before final synthesis; `REPORT_FIDELITY\|compromised` triggers re-assembly from judge verdicts only; two failures → `"Audit compromised — report re-assembled from verdicts only"` label. |
| Falsifiability drop (not downgrade) | ❌ no | deep-qa's nitpick filter downgrades unfalsifiable concerns to minor notes rather than dropping them. Intentional divergence — user chose to keep this behavior when adopting the other three mechanisms. See `_shared/adversarial-judging.md` §4 for the pattern this skill deliberately departs from. |

## Ensemble Judge Panel (variant-specific)

This section is additive to baseline deep-qa. All baseline execution-model contracts still apply.

### Panel composition

Each judge batch — pass 1 (blind) AND pass 2 (informed) — spawns **three independent judges in parallel**:

| Judge ID | Model | Spawn mechanism | Rationale |
|---|---|---|---|
| `judge_claude` | `sonnet-4.6` | Task tool (general-purpose subagent, `run_in_background=true`) | Claude sibling — one generation/size removed from the coordinator's Haiku baseline; retains Anthropic's critic strengths |
| `judge_openai` | `gpt-5.4` (via `mcp__pal__chat`) | Task tool subagent whose only action is `mcp__pal__chat({model: "gpt-5.4", prompt: <judge_prompt>, working_directory_absolute_path: "/root"})` | OpenAI provider — different training/RL → decorrelated failure modes |
| `judge_gemini` | `gemini-2.5-pro` (via `mcp__pal__chat`) | Task tool subagent whose only action is `mcp__pal__chat({model: "gemini", prompt: <judge_prompt>, working_directory_absolute_path: "/root"})` | Google provider — third lineage; 1M context not strictly needed here but adds independence |

**Why Task-wrap the pal calls:** preserves `run_in_background=true`, Monitor-based watchdog, and the per-spawn state.json contracts from baseline. The subagent's prompt is: "Call `mcp__pal__chat` with model={M}, prompt=<verbatim judge prompt from SYNTHESIS.md>, write the structured output to {output_path}. Do not add commentary. Return when the write completes." One tool call, deterministic wrapper.

### Per-judge output paths

For pass 1: `deep-qa-{run_id}/judges/batch_{round}_{batch_num}_pass1_{judge_id}.md`
For pass 2: `deep-qa-{run_id}/judges/batch_pass2_{batch_num}_{judge_id}.md`

Each judge writes the same structured `DEFECT_ID / SEVERITY / CONFIDENCE / RATIONALE[ / CALIBRATION]` block format defined in SYNTHESIS.md — one file per (batch × judge). The panel of three produces three files per batch.

### Aggregation rule (applied after batch drains)

Aggregation is **deterministic and externalized** to `aggregate_ensemble_judges.py` (in this skill's directory). The coordinator invokes it via Bash rather than computing vote tallies inline — this removes coordinator-arithmetic risk and keeps the semantics auditable + unit-testable.

**Invocation (pass 1):**
```bash
python3 ~/.claude/skills/deep-qa-ensemble-v1/aggregate_ensemble_judges.py \
  --batch-id {batch_id} \
  --pass 1 \
  --judge-file judge_claude=deep-qa-{run_id}/judges/batch_{round}_{batch_num}_pass1_judge_claude.md \
  --judge-file judge_openai=deep-qa-{run_id}/judges/batch_{round}_{batch_num}_pass1_judge_openai.md \
  --judge-file judge_gemini=deep-qa-{run_id}/judges/batch_{round}_{batch_num}_pass1_judge_gemini.md \
  --out-json deep-qa-{run_id}/judges/batch_{round}_{batch_num}_pass1_aggregated.json \
  --out-summary-md deep-qa-{run_id}/judges/batch_{round}_{batch_num}_pass1_aggregated.md
```

**Invocation (pass 2):** same shape, but `--pass 2` and add `--prior-pass1-json path/to/pass1-aggregated.json` so the aggregator applies the forced-consistency calibration override.

**Rules the script applies (canonical — do not duplicate inline):**

1. **Severity vote:** `critical > major > minor`. ≥2 agree → that severity. 3-way split → max (fail-safe up). Partial panel of 2 → agreement or max. Partial panel of 1 → use it, flag partial. 0 parseable → retain critic-proposed severity (pass 1) or prior pass-1 aggregated severity (pass 2).
2. **Confidence:** max across judges whose verdict matched the aggregated severity.
3. **Calibration (pass 2):** vote + `upgrade` on 3-way split. **Forced-consistency override** then applied: aggregated pass-2 severity > pass-1 aggregated → `upgrade`; less → `downgrade`; equal → `confirm`. Override wins over vote when they disagree, and the override is logged in `notes`.
4. **Rationale:** concatenation of all parseable judges: `[{judge_id}/{model}] ...` joined with ` | `.
5. **Agreement rate:** `n_matching_majority / n_parseable`.
6. **All-fail handling:** batch recorded as `aggregation_status: "failed"`. Exit code is still 0; check stderr for `batch ... failed=N partial=N` summary.

**The aggregator's output JSON is the authoritative record** — the coordinator reads it to populate `defects.{id}.judge_pass_{N}_verdict` fields in state.json. Coordinator is NOT permitted to recompute severity/calibration from the per-judge files directly.

### State schema extensions (additive — baseline STATE.md still valid)

`background_tasks.judges[]` entries add a `judges` sub-array per batch:

```json
{
  "batch_id": "batch_1_1",
  "defect_ids": ["defect_001", "defect_002"],
  "round": 1,
  "pass": 1,
  "status": "running|completed|partial|timed_out|failed",
  "judges": [
    {"id": "judge_claude", "model": "sonnet-4.6", "provider": "anthropic", "task_id": "bg_...", "status": "running|completed|timed_out", "output_path": "..."},
    {"id": "judge_openai", "model": "gpt-5.4", "provider": "pal-custom", "task_id": "bg_...", "status": "running|completed|timed_out", "output_path": "..."},
    {"id": "judge_gemini", "model": "gemini-2.5-pro", "provider": "pal-custom", "task_id": "bg_...", "status": "running|completed|timed_out", "output_path": "..."}
  ],
  "aggregation_log_path": "deep-qa-{run_id}/judges/batch_1_1_pass1_aggregated.md"
}
```

`defects.{id}.judge_pass_{N}_verdict` adds two fields:

```json
"judge_pass_1_verdict": {
  "severity": "major",
  "confidence": "high",
  "rationale": "[sonnet-4.6] ... | [gpt-5.4] ... | [gemini-2.5-pro] ...",
  "agreement_rate": 0.67,
  "per_model": [
    {"model": "sonnet-4.6", "severity": "major", "confidence": "high", "rationale": "..."},
    {"model": "gpt-5.4",    "severity": "major", "confidence": "medium", "rationale": "..."},
    {"model": "gemini-2.5-pro", "severity": "minor", "confidence": "low", "rationale": "..."}
  ]
}
```

Pass 2 additionally stores `calibration` (aggregated) and `per_model[].calibration`.

### Canonical pal-subagent prompt (paste VERBATIM)

When spawning `judge_openai` or `judge_gemini`, use the Task tool with a general-purpose subagent and pass this exact prompt (with `{MODEL}`, `{BATCH_INPUT_PATH}`, `{OUTPUT_PATH}`, and `{PASS_N_JUDGE_PROMPT}` filled in). **Do not paraphrase** — the subagent's reliability depends on a tight, well-tested wrapper prompt.

```
You are a wrapper agent. Your single job is to call one MCP tool and write its raw response to a file. You MUST NOT critique, summarize, reformat, or add commentary.

Execute these steps in order:

1. Call `mcp__pal__chat` with these exact arguments:
   - model: "{MODEL}"
   - prompt: |
{PASS_N_JUDGE_PROMPT}

   - working_directory_absolute_path: "/root"

2. Extract the `content` field from the tool response (ignore `metadata`, `continuation_offer`, `content_type`).

3. Use the Write tool to save the extracted content VERBATIM to: {OUTPUT_PATH}
   Do not edit it. Do not add a header. Do not fix formatting. If the judge model's output doesn't match the expected DEFECT_ID/SEVERITY/CONFIDENCE format, the aggregator script handles that — your job is only to preserve the raw response.

4. Return a single line: `JUDGE_WRITE_OK {OUTPUT_PATH}` if successful, or `JUDGE_WRITE_FAILED {reason}` if any step failed.

Failure modes to handle:
- If `mcp__pal__chat` raises an error, do NOT retry. Write a placeholder file with the literal content `JUDGE_CALL_FAILED: {error_message}` to {OUTPUT_PATH} and return `JUDGE_WRITE_FAILED pal_call_error`. The aggregator treats this as `missing_file`.
- If the tool response is missing a `content` field, write `JUDGE_PARSE_FAILED: response had no content field` to {OUTPUT_PATH} and return `JUDGE_WRITE_FAILED no_content`.

Do not call any other tools. Do not write to any other path. Do not spawn any sub-tasks.
```

**`{PASS_N_JUDGE_PROMPT}` substitution:** paste the literal pass-1 or pass-2 judge prompt from SYNTHESIS.md verbatim, with `{batch_input_path}` replaced by `{BATCH_INPUT_PATH}`. Do not modify the judge prompt itself — the ensemble protocol depends on all three judges seeing the same instructions.

**`judge_claude` uses a different template** — it is a Task subagent running Sonnet natively (not via pal). Its prompt is just the pass-N judge prompt directly, with an added line `Write your output verbatim to: {OUTPUT_PATH}` at the end. No MCP wrapping needed.

---

### Post-hoc analysis artifact

After all judge batches drain, write `deep-qa-{run_id}/judges/ensemble-summary.md` containing:
- Global agreement rate (across all defects, all batches, both passes)
- Per-pair agreement (sonnet↔gpt, sonnet↔gemini, gpt↔gemini) — surfaces which provider pairs decorrelate most
- Disagreement cases with the largest severity span (e.g., `critical` vs `minor`) — worth human review
- Per-model confidence distribution — are any models systematically over/under-confident?

This file is read by skill-bench's post-run parser (via the schema above) and feeds the benchmark.json `run_summary`.

---

## Artifact Types

| Type | Applies to | Required QA Categories |
|------|-----------|------------------------|
| `doc` | specs, design docs, RFCs, API docs, architecture docs | completeness, internal_consistency, feasibility, edge_cases |
| `code` | source code, system architecture descriptions | correctness, error_handling, security, testability |
| `research` | research reports, literature reviews, deep-research outputs | accuracy, citation_validity, logical_consistency, coverage_gaps |
| `skill` | Claude skills, system prompts, agent specs, tool instructions | behavioral_correctness, instruction_conflicts, injection_resistance, cost_runaway_risk |

See DIMENSIONS.md for full dimension tables and angle examples.

---

## Workflow

### Phase 0: Input Validation Gate

#### `--diff` mode (fast post-commit QA)

When `--diff [ref]` is present, the artifact is built from the git diff rather than a full file. This costs ~10% as much as a full-repo QA and catches regressions in the changed code and its immediate callers.

**Step 0a-diff — Build diff artifact:**

1. Run `git diff {ref}` — include ALL tracked files, not just `*.py`. Frontend code (`.svelte`, `.tsx`, `.ts`, `.js`, `.vue`), templates, SQL, proto files, YAML manifests, and JSON fixtures are routinely consumers of data contracts that Python code changes, and must be in scope. (Previously this step filtered to `*.py`, which caused UI/frontend defects in multi-language projects to be invisible to diff-mode QA.)
2. If diff is empty: error "No changes found between HEAD and {ref}."
3. Extract changed files from the diff header lines (`--- a/...`, `+++ b/...`).
4. For each changed file, find **callers** of any added/modified function or method:
   - Use `grep -rn "def <name>" <changed_file>` to extract function names from `+` lines
   - Use `grep -rn "<name>(" --include="*.py"` to find call sites in the repo
   - Include the call-site file + ±10 lines of context for each hit (cap at 5 callers per function, 20 functions total)
5. Build `artifact.md` with three sections:
   ```
   # Diff QA Artifact
   ## Ref: {ref} → HEAD
   ## Changed files: {list}

   ## Section 1: The Diff
   {full git diff output, unified format}

   ## Section 2: Caller Context
   {per-function: function name, callers found, relevant snippets}

   ## Section 3: Pre-existing behavior (for context only)
   {unchanged surrounding code ±20 lines for each changed hunk}
   ```
6. **Size check:** same as normal mode (~80k token warning).

**Automatic angle seeding in diff mode** — in addition to normal dimension angles, always add these high-priority angles before round 1:
- "**Legacy-symbol sweep (MANDATORY, CRITICAL priority).** Enumerate every pre-change name, string literal, dict key, constant, attribute, env var, or magic value that this PR is *replacing* (e.g. old function names, hardcoded strings like `\"start\"`/`\"end\"`, old config keys, old sentinels). Then `grep -rn` the **entire repository** — NOT just changed files — for each one. For every remaining occurrence, classify as: (a) correctly updated in this PR, (b) legitimate legacy-compat path with a documented fallback, (c) stale docstring/comment referencing the old contract, or (d) a MISSED UPDATE. Report every (c) and (d). This angle must spawn its own critic; do not fold it into another dimension. Missed updates in unchanged files are the highest-impact latent defects and they are invisible to diff-scope review."
- "**Contract fanout audit (MANDATORY, CRITICAL priority whenever the PR changes ANY named or shared contract).** A contract here is anything that connects code that changes to code that doesn't: an API signature, a data-shape (dict keys, schema fields, enum values, wire format), a calling convention (how a command is invoked, how a process re-enters itself, how a handler is registered), a named symbol used as an identifier, a protocol, or a configuration key. For every changed contract: (1) enumerate every **producer** of the contract — every place that emits, constructs, serializes, or writes the contract value; (2) enumerate every **consumer** — every place that parses, reads, introspects, or renders it; (3) scope the search across the ENTIRE artifact surface — every file, every language, every format the repo contains — not just the files the diff touches and not just the language the diff is written in; (4) for each location classify as: correctly updated / legitimate legacy-compat with documented fallback / stale docstring-comment-message / MISSED UPDATE, and report every stale and missed as separate defects. This principle specializes to many concrete patterns depending on the contract — for illustration only: if the contract is how subprocesses re-enter an entrypoint, producers are every command-builder across orchestrators/runtimes/sidecars/CLI-wrappers; if the contract is the shape of a persisted artifact, consumers include every reader across languages (frontend UIs, generated types, schemas, views, fixtures, dashboards); if the contract is a named identifier used as a dict key, both producers and consumers span the repo. The failure mode being defended against is that refactors reliably update the two or three closest-to-hand producers and consumers and miss the rest — that set of 'the rest' is where the highest-leverage latent defects live. Adapt the concrete search to the actual contract under review; the invariant is the breadth, not any specific command. Reasonable starting points are grep-based symbol searches, git-log-based caller traces, and type/schema-tool searches; if the artifact has non-text components (binaries, generated files), note the gap explicitly rather than skipping."
- "**Docstring / comment contract consistency.** Grep for docstrings, inline comments, and user-facing error messages that reference the OLD contract by name. For a PR making a backward-compat claim, docstrings that still say `run['end']` when end steps can now be renamed ARE user-facing defects — they mislead readers who trust documented contracts. File as minor but DO file."
- "For every new conditional expression in the diff (`if`, `elif`, `while`): what are the False/empty/None/zero branches? Are they all safe?"
- "For every changed function signature or return type: do all callers handle the new contract?"
- "For every attribute, return value, or method that the PR newly makes `Optional[X]` (previously always non-None): grep every consumer in the repo and verify each handles `None` without crash, `KeyError`, or silent wrong-behavior. Do not assume lint or validation catches it earlier — audit the consumer's code."
- "For every new subprocess, file handle, network connection, or lock opened in the diff: is it always closed/drained/released on all exit paths?"
- "For every security-sensitive path touched (auth, subprocess args, file paths, serialization): what are the injection/bypass edge cases introduced by the change?"

**`artifact_type` in diff mode:** Default to `code`. Override with `--type` as normal.

**`--diff` + `--auto`:** Fully unattended — runs with defaults, no gates.

**Print:** `Starting deep QA on: diff {ref}..HEAD ({N} files changed) [type: code] [run: {run_id}]`

After building the diff artifact, proceed to Phase 1 as normal. The artifact IS the diff + context; the workflow is identical.

---

**Step 0a — Read artifact (normal mode, when `--diff` is NOT present):**
- If argument is a file path: read file, write contents to `deep-qa-{run_id}/artifact.md`
- If argument is inline content: write to `deep-qa-{run_id}/artifact.md` ⚠️ inline content is silently truncated at context limit — warn user if content appears large
- If empty or inaccessible: error
- **Size check:** After writing, check approximate token count. If artifact.md exceeds ~80k tokens: warn "Artifact is large (~{N} tokens). Haiku critics (depth 2+) may only see part of it." If `--auto`: proceed with warning. If interactive: ask "Continue? [y/N]"

**Step 0b — Artifact type detection:**
- **If `--type` is provided:** use it as authoritative, skip content inference, skip ambiguity prompt. Store `artifact_type = --type` in state.json.
- **If `--type` not provided:** infer from content (file headers, structure, terminology — see DIMENSIONS.md). If ambiguous: "I'm interpreting this as type=[X] because [2-3 evidence signals]. Correct? [y/N/type:doc|code|research|skill]" (skip prompt if `--auto`).
- Store `artifact_type` in state.json.

**Multi-file discovery (after type is determined):**
- If `artifact_type == "skill"` AND argument was a file path: check the parent directory for companion files matching `DIMENSIONS.md`, `FORMAT.md`, `STATE.md`, `SYNTHESIS.md`.
- If companion files exist: concatenate all into `artifact.md` (SKILL.md first, then companions alphabetically). Show `Files in scope: [list]` in the pre-run scope declaration.
- If `--auto` and companions found: include them automatically.

**Step 0c — Safety check:**
- If artifact contains credentials, tokens, or PII: warn user before passing to subagents
- If artifact requests harmful functionality: decline

**Print:** `Starting deep QA on: {artifact_name} [type: {artifact_type}] [run: {run_id}]`

---

### Phase 1: Dimension Discovery (see DIMENSIONS.md)

- Select QA dimensions from DIMENSIONS.md based on `artifact_type`
- Generate 2-4 critique angles per dimension + 2-3 cross-dimensional angles
- Required categories per type must each get at least one angle (CRITICAL priority if uncovered after round 1)
- Cap frontier at 30 angles total

**Pre-run scope declaration (show before proceeding; skip entirely if `--auto`):**
```
Deep QA: "{artifact_name}"
Artifact type: {artifact_type}
Files in scope: {list of files included in artifact.md}
QA dimensions ({N}): {list}
Initial angles: {count}
Suggested max_rounds: {recommendation}
Hard stop: {recommendation * 2} rounds (non-overridable)
Wall-clock estimate: {time range}
Invocation: {interactive | automated (--auto)}

Set max_rounds [default {recommendation}]: _
Continue? [y/N]
```
If `--auto`: use recommended max_rounds automatically, do not show this prompt.

**max_rounds recommendation formula:**
```
initial_angles = count of angles in Phase 1
min_rounds = ceil(initial_angles / 6)     # 6 agents/round
recommended = ceil(min_rounds * 1.3)      # 30% expansion from agent-discovered sub-angles
recommended = max(recommended, 3)         # never suggest < 3 rounds
recommended = min(recommended, 6)         # cap at 6 for typical artifacts
```

---

### Phase 2: Initialize State

- Generate run_id: `$(date +%Y%m%d-%H%M%S)`
- Create directory structure:
  - `deep-qa-{run_id}/state.json` — run state (see STATE.md)
  - `deep-qa-{run_id}/critiques/` — one file per critique angle
  - `deep-qa-{run_id}/angles/` — per-angle input files for critics
  - `deep-qa-{run_id}/judge-inputs/` — per-defect input files for severity judges
  - `deep-qa-{run_id}/artifact.md` — copy of artifact content
  - `deep-qa-{run_id}/qa-report.md` — written at Phase 6
- Write lock file: `deep-qa-{run_id}.lock` — verify write succeeded before proceeding
- Store `hard_stop = max_rounds * 2` in state.json — immutable after initialization

---

### Phase 3: QA Rounds

**Hard stop check (fires BEFORE prospective gate, every round, unconditionally):**
```
if current_round >= state.hard_stop:
    → terminate immediately; no prompt; no extension offered
    → label: "Hard stop at round {hard_stop}"
    → proceed directly to Phase 4/Phase 5.5/Phase 6
```
This check cannot be bypassed. Extensions update `max_rounds` but never `hard_stop`.

**Prospective gate (fires after hard stop check; skipped if `--auto`):**
```
About to run QA Round {N}: {frontier_size} angles queued
Critics this round: up to 6 | Potential judge agents: up to {frontier_pop × 5}
Estimated cost: ~${critics_cost + judges_cost + summary_cost} ({running_total} spent so far)
Continue? [y/N/redirect:<focus>]
```
- If N: stop → label: `"User-stopped at round N"`
- If redirect: add high-priority angle targeting the specified focus, then proceed
- Skip if `--auto`

**Per round:**
1. Pop up to `max_agents_per_round` (6) highest-priority angles from frontier; enforce frontier cap (see STATE.md)
2. Write all required data to files BEFORE spawning, then **verify each write** (file exists + non-empty):
   - Known defects file: `deep-qa-{run_id}/known-defects.md`
   - Angle files: `deep-qa-{run_id}/angles/{angle.id}.md`
   - If any verification fails: halt with error, do not spawn
3. For each angle, write `status: "in_progress"` and `spawn_time_iso` to state.json **BEFORE** calling Agent tool. After writing, **re-read state.json and verify `generation == N+1`**. If mismatch: log conflict, retry once with fresh read, then halt.
4. Spawn critic agents in parallel (120s timeout)
5. On timeout: mark `timed_out`, write `"generation": += 1`, do NOT re-queue, do NOT increment dedup counter
6. Collect new angles from ALL completed agents BEFORE running dedup
7. Apply dedup against stable pre-round snapshot. **Assign `depth = parent.depth + 1`** to each critic-reported angle. Reject angles where `depth > max_depth`. Enforce frontier cap with required-category protection (see STATE.md).
8. For each new defect: **Dimension cross-check (synchronous):** verify the critique file's declared `**QA Dimension:**` header matches the angle's assigned dimension in state.json. If mismatch: flag as potential injection, do NOT set `required_categories_covered.{category}` true. Create defect in state.json with critic-proposed severity and `judge_status: "pending"`.
9. Run coverage evaluation: read `required_categories_covered` from **state.json** (not coordinator-summary.md). For any uncovered required category: generate CRITICAL-priority angle. Write `"generation": += 1` after updating `coverage_gaps` and `rounds_without_new_dimensions`.
10. ▶ **Background severity judges (pass-1 blind) — ENSEMBLE PANEL:** Batch new defects into groups of up to 5. For each batch: write combined defect data to `deep-qa-{run_id}/judge-inputs/batch_{round}_{batch_num}.md` **with the critic-proposed severity STRIPPED from each defect entry** (blind-severity protocol; see [`_shared/adversarial-judging.md`](../_shared/adversarial-judging.md) §1). Then spawn a **three-model judge panel in parallel** — all with `run_in_background=true`:

   1. **`judge_claude`** — Task tool spawn of a general-purpose subagent running Sonnet 4.6 (project Claude). Prompt is the pass-1 blind judge template from SYNTHESIS.md with `{batch_input_path}` and instruction to write the structured output to `deep-qa-{run_id}/judges/batch_{round}_{batch_num}_pass1_judge_claude.md`.

   2. **`judge_openai`** — Task tool spawn of a general-purpose subagent whose sole task is: `mcp__pal__chat({model: "gpt-5.4", prompt: <verbatim pass-1 blind judge prompt with {batch_input_path} resolved>, working_directory_absolute_path: "/root"})` — then write the returned text verbatim to `deep-qa-{run_id}/judges/batch_{round}_{batch_num}_pass1_judge_openai.md`. Subagent prompt explicitly forbids commentary, JSON reformatting, or summarization — preserve the pal response as-is.

   3. **`judge_gemini`** — identical to `judge_openai` but with `model: "gemini"` (alias for gemini-2.5-pro) and output to `...pass1_judge_gemini.md`.

   Record the batch in `background_tasks.judges` in state.json with `pass: 1` and the extended `judges[]` sub-array (see "State schema extensions" in the Ensemble Judge Panel section). Each entry captures `{id, model, provider, task_id, status, output_path}` for all three spawns.

   Aggregated `judge_pass_1_verdict` fields are populated during Phase 5.5.a drain — NOT at spawn time.

   **Watchdog (per-judge):** each of the three spawns is armed with a Monitor-based staleness watchdog per [`_shared/subagent-watchdog.md`](../_shared/subagent-watchdog.md) §"Flavor A" with `STALE=3 min, HUNG=10 min`. If any single judge watchdog fires, its output path is marked `timed_out_heartbeat` in `background_tasks.judges[].judges[]` — the remaining two continue. If two or more fire, the batch is flagged `partial` and aggregation handles it per the "partial results" branch in the aggregation rule.
11. **Background coordinator summary:** Spawn Haiku subagent with `run_in_background=true` to write a **cumulative** coordinator summary (see SYNTHESIS.md). Record in `background_tasks.summaries` in state.json.
12. Increment round → **immediately proceed to next round's step 1** (do not wait for background tasks)

**Pipelining rationale:** Severity judges and coordinator summaries are reporting artifacts consumed only in Phase 6. They do not affect angle selection, dedup, or coverage evaluation. Running them in the background while the next round's critics execute hides their latency entirely.

**No redesign phase.** Defects are catalogued with severity; status remains `open` unless disputed by validation.

---

### Phase 4: Fact Verification (research artifacts only)

For `artifact_type == "research"`, run before final synthesis. Skip entirely for other types.

- Spawn Haiku verification agent
- Extract top-N factual claims (N = min(20, total claims found))
- Risk-stratified sampling: single-source primary → numerical/statistical → contested → corroboration candidates
- Spot-check citation URLs: accessible? attributed claim present in source text?
- For numerical claims: compare EXACT numbers — flag mismatch even if semantically similar
- Output: `deep-qa-{run_id}/verification.md`
- See SYNTHESIS.md for full protocol

---

### Phase 5: Termination Check

**Note:** The hard stop check at the start of Phase 3 fires unconditionally before this check is evaluated. All labels below apply to paths that reach Phase 5.

**Any-of-4 — evaluate in order, stop when FIRST is true:**
1. **User-stopped:** User chose N at a prospective gate → label: `"User-stopped at round N"`
2. **Coverage plateau:** `rounds_without_new_dimensions >= 2` AND all explored angles in state.json have `exhaustion_score >= 4` → label: `"Coverage plateau — frontier saturated"`
3. **Budget soft gate:** `current_round >= max_rounds` with non-empty frontier. Show gate (skip if `--auto`):
   ```
   Budget limit reached (max_rounds={N}). Frontier still has {M} unexplored angles.
   Hard stop at round {hard_stop} — remaining headroom: {hard_stop - current_round} rounds.
   Options: [y] Extend by {min(recommended, hard_stop - current_round)} more rounds  [+N] Custom  [n] Stop
   ```
   - Extension validation: cap any extension at `hard_stop - current_round`; reject extensions that would reach or exceed `hard_stop`
   - User chooses n → label: `"Max Rounds Reached — user stopped"`
   - User extends → update `max_rounds`, continue; `hard_stop` is NOT updated
   - `--auto`: stop immediately → label: `"Max Rounds Reached"`
4. **Frontier empty:** evaluate "Conditions Met" check below

**"Conditions Met" check (only when condition 4 fires):**
- Read `required_categories_covered` from **state.json**
- ALL three must be true: (1) frontier empty, (2) all required categories covered, (3) `rounds_without_new_dimensions >= 2`
- All true → label: `"Conditions Met"`
- Any false → label: `"Convergence — frontier exhausted before full coverage"` (list uncovered required categories in report)

**Complete label vocabulary — all reachable paths:**
| Label | When |
|-------|------|
| `"Conditions Met"` | Condition 4 fires + all-3 satisfied |
| `"Coverage plateau — frontier saturated"` | Condition 2 |
| `"Max Rounds Reached — user stopped"` | Condition 3 + user n |
| `"Max Rounds Reached"` | Condition 3 + --auto |
| `"User-stopped at round N"` | Condition 1 |
| `"Convergence — frontier exhausted before full coverage"` | Condition 4 + not all-3 |
| `"Hard stop at round N"` | Phase 3 pre-check fires |
| `"Audit compromised — report re-assembled from verdicts only"` | Phase 5.6 rationalization auditor reports `REPORT_FIDELITY\|compromised` on two consecutive assemblies |

Never use a label not in this table. Never write "no defects remain."

---

### Phase 5.5: Drain Background Tasks + Pass-2 Informed Severity

Before proceeding to Phase 5.6 (rationalization audit) and Phase 6 (final report), all background tasks from the pipelined rounds must complete, AND the blind-severity protocol must finish pass 2.

**5.5.a — Drain pass-1 blind judges:**

1. ▶ **Wait for ALL per-judge spawns across all batches.** For each entry in `background_tasks.judges[]` where `pass == 1`, iterate its `judges[]` sub-array and call `TaskOutput` with `block=true` on each `task_id`. A batch is considered drained when every one of its three per-judge spawns has either completed or timed out.
2. **Wait for ALL background coordinator summaries** (from `background_tasks.summaries`). Use `TaskOutput` with `block=true` for each.
3. ▶ **Aggregate pass-1 judge results across the three-model panel. For each batch:**
   a. Read each of the three per-judge output files (`batch_{round}_{batch_num}_pass1_judge_{id}.md`). Parse structured `DEFECT_ID` / `SEVERITY` / `CONFIDENCE` / `RATIONALE` blocks from each.
   b. If a file is missing, empty, or unparseable, mark that judge as `timed_out` or `unparseable` in `background_tasks.judges[].judges[].status`. Do NOT halt — proceed with the remaining judges and apply the partial-results rule from "Aggregation rule" §1.
   c. For each defect in the batch, collect up to 3 per-judge verdicts and apply the aggregation rule (severity vote + max confidence + concat rationale, see "Ensemble Judge Panel / Aggregation rule"). Store the aggregated result as `defects.{id}.judge_pass_1_verdict` with the `per_model` sub-array populated from the three parsed verdicts (including `timed_out` stubs for missing ones).
   d. Write the aggregation summary to `deep-qa-{run_id}/judges/batch_{round}_{batch_num}_pass1_aggregated.md` for post-hoc inspection — one section per defect showing all 3 per-judge lines + the aggregated line + agreement rate.
   e. Set `defects.{id}.judge_status: "pass_1_completed"`.
   f. Set `background_tasks.judges[].status`: `completed` if all 3 judges parseable, `partial` if 1–2 timed out, `failed` if all 3 timed out.
   g. Write `"generation": += 1` after each batch is processed.
4. ▶ **Handle panel-level pass-1 failures:** A defect whose batch aggregated as `failed` (all 3 judges timed out/unparseable) retains critic-proposed severity. Set `judge_status: "pass_1_timed_out"`. Log `JUDGE_PANEL_TOTAL_FAILURE_PASS_1: {batch_id} defect_ids={...}`. A batch that aggregated as `partial` uses the aggregation rule's partial-results branch and proceeds normally to pass 2 — no special handling.

**5.5.a-coherence — Cross-finding coherence integrator (fires after pass-1 drain, before pass-2):**

Per [`_shared/cross-finding-coherence.md`](../_shared/cross-finding-coherence.md):

1. Collect all parseable critic output files from all rounds (post-dedup).
2. Write integrator input manifest to `deep-qa-{run_id}/coherence/input-manifest.md` containing: list of all critic output file paths, artifact path, known-defects path, dimension taxonomy.
3. Spawn Sonnet coherence-integrator agent with the manifest. Output: `deep-qa-{run_id}/coherence/round-all-coherence.md`. Timeout: 120s.
4. Parse `STRUCTURED_OUTPUT` block for `FINDING|`, `GAP|`, and `PATTERN|` lines.
5. For each `FINDING|{id}|{annotation}` line: attach annotation to `defects.{id}.coherence_annotation` in state.json. Write `generation += 1`.
6. For each `GAP|{dim_a}|{dim_b}|{description}|{angle}` line: create a CRITICAL-priority angle in the frontier for the next round (if rounds remain) or flag in the final report (if this is the last round).
7. For each `PATTERN|{pattern_id}|{finding_ids}|{root_cause}|{severity_suggestion}` line: store in `state.json.emergent_patterns[]` for Phase 6 final report.
8. If integrator output is unparseable or timed out: log `COHERENCE_PARSE_FAILED` or `COHERENCE_TIMED_OUT`. Proceed without annotations — pass-2 judges run normally (degraded mode). Flag in Phase 6 report.
9. If `STANDALONE` rate is 100% across 6+ findings: log `COHERENCE_SHALLOW` warning — include in Phase 6 caveats.

**Why between pass-1 and pass-2:** Pass-1 judges classify severity blind (without critic or cross-finding context). The integrator runs on critic output, not judge output — it identifies relationships between findings, not between severities. Pass-2 judges then receive BOTH the pass-1 verdict AND the coherence annotation, allowing them to upgrade severity for pattern-members or scrutinize contradicted findings.

**5.5.b — Spawn pass-2 informed judges (ENSEMBLE PANEL):**

For each defect where `judge_pass_1_verdict` exists:
5. Write `deep-qa-{run_id}/judge-inputs/batch_pass2_{batch_num}.md` containing: the full defect (INCLUDING critic-proposed severity this time), the **aggregated** pass-1 verdict (from the three-judge panel; include the aggregated severity + agreement_rate + a compact per-model summary line so the pass-2 panel sees how much the pass-1 panel diverged), any coherence annotation from Phase 5.5.a-coherence (contradiction/pattern/standalone status), and the pass-2 prompt asking the judge to confirm, upgrade, or downgrade with rationale. Coherence annotations give the judge cross-finding context: a `PATTERN_MEMBER` annotation suggests the judge should consider aggregate severity; a `CONTRADICTS` annotation suggests the judge should scrutinize the finding's evidence base.
6. ▶ **Spawn three pass-2 judges in parallel** — same panel composition as pass 1 (`judge_claude` Sonnet 4.6 via Task tool, `judge_openai` GPT-5.4 via `mcp__pal__chat`

…(truncated)