Deep Debug Skill — Ensemble Variant v1
Benchmark variant. Structurally identical to
deep-debugexcept for Phase 3 step 3b (spawn) and step 3e (drain + aggregate). Both phases now spawn a three-model judge panel per batch — Sonnet 4.6 viaTask+ GPT-5.4 viamcp__pal__chat+ Gemini 2.5 Pro viamcp__pal__chat— with explicit plausibility aggregation viaaggregate_ensemble_judges_debug.py. All other phases (Phase 3a, 3a-coherence, 3c, 3d, 3f critic/rebuttal, Phases 4/5/6/7/8) unchanged from baseline.Panel composition: same 3 providers as deep-qa-ensemble-v1. See that skill's Ensemble Judge Panel section for the full rationale — panel composition, canonical pal-subagent prompt template, and watchdog thresholds are identical.
Aggregation: majority vote on
PLAUSIBILITY. Tie-break on 3-way split isdisputed(NOT fail-up) — promoting unvetted hypotheses toleadingon disagreement would feed weak hypotheses into Phase 4 probe generation. Evidence tier = min across parseable judges (most conservative). Pass-2 verdict = majority vote, default toCONFIRMon tie. All per-model verdicts preserved in state.Aggregator script:
aggregate_ensemble_judges_debug.pyin this skill's directory. Invoke via Bash after draining each batch. See the invocation example in step 3e below.
Adversarial hypothesis-driven debugging. Given a bug, generate competing hypotheses across orthogonal dimensions, judge each independently (two-pass blind), run discriminating probes that falsify the weaker, fix the survivor with test-first discipline, and escalate to architectural review after 3 failed attempts. Output is a verified fix with a failing-test-now-passes proof, OR an honest termination label naming why the bug resisted a code-level fix.
Execution Model
All operations use Claude Code primitives. These contracts are non-negotiable:
- All data passed to agents via files, never inline. Evidence, hypothesis lists, probe specs, known-hypothesis IDs — all written to disk before the agent prompt. Inline data is silently truncated.
- State written before agent spawn, not after.
spawn_time_isois written to state.json before the Agent tool call. Spawn failure recordsspawn_failedstatus. Resume uses persisted state, never in-memory reconstruction. - Structured output is the contract; free-text is ignored. Every judge, probe-verdict, and validator produces machine-parseable structured lines. Coordinator reads only structured fields. Unparseable output triggers fail-safe classification (
disputed, neverrejected). Hypothesis output files MUST containSTRUCTURED_OUTPUT_START/STRUCTURED_OUTPUT_ENDmarkers; files without these markers are treated as failed. - No coordinator self-review of anything load-bearing. Hypothesis plausibility, evidence tier, rebuttal verdicts, probe winner classification — all delegated to independent agents. The coordinator orchestrates; it does not evaluate.
- Termination labels are honest. One of 7 defined labels. Never "probably fixed," never "looks right," never "no critical bugs remain." A fix is
Fixed — reproducing test now passesonly when the failing test exists, passes with the change, and the full suite is clean. - Hard ceilings are absolute.
max_cycles = 3,max_probes_per_cycle = 3,fix_attempt_count ≤ 3. Three fix attempts is evidence the hypothesis space is wrong — Phase 7 architectural escalation is mandatory, never optional.
Shared contracts: this skill inherits the four execution-model contracts (files-not-inline, state-before-agent-spawn, structured-output, independence-invariant) from _shared/execution-model-contracts.md. The items listed above are the skill-specific elaborations; the shared file is authoritative for the base contracts.
Cross-finding coherence: this skill applies the coherence-integrator pattern from _shared/cross-finding-coherence.md at Phase 3, after hypothesis agents complete and BEFORE the judge classifies plausibility. The integrator reads all hypothesis files simultaneously and annotates each hypothesis with cross-hypothesis relationships (contradictions indicating the same root cause viewed from different angles, emergent patterns suggesting a higher-level architectural issue, coverage gaps in the 8-dimension space). These annotations are included in judge input files.
Subagent watchdog: every run_in_background=true spawn (hypothesis agents, judges, evidence-gatherer, architect) MUST be armed with a staleness monitor per _shared/subagent-watchdog.md. Use Flavor A with thresholds STALE=5 min, HUNG=20 min for Sonnet hypothesis agents; STALE=3 min, HUNG=10 min for Haiku judges and evidence-gatherer. Debugging agents that hang silently are the exact failure mode this skill is meant to prevent — applying it to the skill itself is load-bearing. Contract inheritance: timed_out_heartbeat joins this skill's 7-label termination vocabulary at the per-lane level (hypothesis agent / judge / probe); stalled_watchdog and hung_killed join hypotheses.{id}.status. A watchdog-killed hypothesis lane never returns leading or rejected — its verdict is absent, not assessed.
Philosophy
Debugging fails most often in three ways:
- Fixation on the symptom site — the stack trace points at file:line, the fix goes there, the real cause is 5 calls upstream. Solo debuggers rarely trace data-flow all the way back.
- Hypothesis anchoring — you form the first plausible hypothesis, evidence confirms it (because you're looking for confirmation), you fix it, the bug returns.
- Fix-and-retry spiral — 3 failed attempts is the signal the abstraction is wrong, but under time pressure debuggers try fix #4, #5, #6, each one revealing a new problem in a different location.
Deep-debug addresses all three structurally:
- Orthogonal dimensions force at least one hypothesis in
concurrency,environment, andarchitecturecategories — the three most commonly under-investigated - Independent judge with disconfirmation mandate prevents anchoring on the first fit
- Hard architectural escalation at 3 failed fixes converts fix-spiral into a conscious pattern-level decision
If you only take one idea from this skill: evidence-for and evidence-against are the hypothesis contract. A hypothesis without disconfirming searches is never leading; it's plausible at best. See EVIDENCE.md.
The Iron Law
NO FIXES WITHOUT A HYPOTHESIS THAT SURVIVES INDEPENDENT JUDGE + DISCRIMINATING PROBE
If Phase 3 judge has not classified the hypothesis as leading AND Phase 4 probe has not confirmed it against the next-best alternative, you cannot advance to Phase 5. Violating the letter of this process is violating the spirit of debugging.
When to Use
Use deep-debug for:
- Bugs that survived a quick-fix attempt (systematic-debugging already ran and the symptom remains or moved)
- Production incidents requiring post-mortem rigor
- Test failures that cannot be reproduced locally
- Performance regressions where the cause is ambiguous
- Build / deploy / signing / infrastructure failures with multi-layer pipelines
- Behavior divergence between environments (local ✓, CI ✗, or one deployment region only)
- Any bug where ≥ 2 plausible root causes exist and the coordinator is tempted to guess
Don't use for:
- Obvious typos, compiler errors with clear messages, or known-issue-with-known-fix cases → use
superpowers:systematic-debugging - Test flakiness where the target is "is this test flaky, and why specifically?" → use
flaky-test-diagnoser(it's purpose-built) - Bugs the user already has a hypothesis for and just wants implemented → implement directly; save deep-debug for when the cause is genuinely unknown
- Pure code review or static audit → use
deep-qawith--type code
Workflow
Phase 0: Input Validation Gate
Before any cycle begins, validate the symptom. Batch clarifying questions — if multiple surface here, present them in one message as a numbered list. Never serially.
Step 0a — Safety check:
- Symptom describes a bug in the user's code, not a request to attack a third-party system
- If harmful: refuse
Step 0b — Scope check:
- Is the symptom specific enough to investigate? ("the app is slow" is too vague; "POST /api/orders p99 latency jumped from 200ms to 2s after {commit}" is fine)
- If too vague: ask for: exact error (copy-paste the message), reproduction steps (commands that trigger it), affected environment (local/CI/prod), and when it started (timestamp or "since merge X")
Step 0c — Reproduction confirmation:
- Attempt to reproduce immediately (run the provided command; trigger the flow)
- Record status in state:
confirmed(every time),intermittent(1-in-N with rate), orunreproducible(cannot trigger) unreproducible→ user batch-question: can they provide fresh logs, a stack trace, or a less-filtered reproduction?- Still
unreproducibleafter user input → terminate with labelCannot reproduce — investigation blockedand exit cleanly. Do NOT hypothesize against a bug you cannot see.
Step 0d — Symptom locking:
- Extract a ≤200-word symptom statement — the exact error, the surface, when it started, the reproduction command
- Compute
symptom_sha256; store in state.json - This statement is the concept-anchor — every hypothesis is a causal explanation of this symptom. Later phases MUST NOT silently rewrite the symptom to match an emerging hypothesis.
symptom_sha256catches tampering.
Step 0e — Pre-mortem micro-round (blind-spot seeding): Before Phase 2 expansion, spawn 1 Haiku agent with this prompt:
Given the symptom
{symptom}, list 5 concrete ways this debugging investigation could miss the real cause. Cover these angles:
- Wrong dimension — the symptom looks like {apparent dimension} but is actually {other dimension}
- Measurement artifact — the bug might not exist; we could be reading stale/wrong evidence
- Environment assumption — local behavior assumed, production differs
- Framework-contract blindness — something assumed guaranteed that actually isn't
- Architectural drift — this is symptom N of an architectural problem; fixing here won't hold Output to
deep-debug-{run_id}/premortem.mdwith one concrete claim per angle.
Each flagged blind-spot becomes an angle with priority: critical and source: premortem in Phase 2.
Step 0f — Pre-run scope declaration:
Deep debug: "{symptom}"
Reproduction: {confirmed | intermittent (1-in-N) | unreproducible-investigation-blocked}
Dimensions: 8 (code-path, data-flow, recent-changes, environment, framework-contract, concurrency-timing, measurement-artifact, architectural-coupling)
Max cycles: {default 3} | Hard stop: {6} cycles | Budget cap: ${25.00}
Estimated cost: ~$3-8 per cycle (Sonnet hypothesis + Haiku judge + Haiku probes)
Invocation: {interactive | --auto | --hard-mode}
Continue? [y/N]
Skip gate if --auto.
Print: Starting deep-debug on: {symptom_excerpt} [run: {run_id}]
Phase 1: Evidence Gathering
Purpose: Collect the ground-truth artifacts every hypothesis in Phase 2 will be evaluated against. Evidence is tier-ranked per EVIDENCE.md.
Step 1a — Symptom capture (always):
- Exact error text / stack trace / log excerpt — copy verbatim into
deep-debug-{run_id}/evidence.md§Symptom - Timestamp of first occurrence (if known)
- Reproduction command (from Phase 0c)
Step 1b — Recent changes snapshot (always):
git log --since='7 days ago' --oneline -- {affected-paths}→ evidence.md §Recent Changes- Lock file diffs (
package-lock.json/poetry.lock/Gemfile.lock) vs. last-known-good, if known - Config diffs
- Even if this looks unrelated to the symptom, record it.
recent-changesis one of the 8 dimensions — agents will search for the connection.
Step 1c — Boundary instrumentation (when multi-layer):
- If the system has obvious layer boundaries (workflow → build → sign; API → service → DB; frontend → gateway → backend): add instrumentation at each layer to log inputs/outputs
- Run the reproduction once with instrumentation; capture all layer logs into evidence.md §Boundary Instrumentation
- This converts tier-5 stack-position guesses into tier-2 primary-artifact evidence for Phase 2
- Skip if single-layer or instrumentation would take > 30 minutes (add to Phase 4 probe list instead)
Step 1d — Lock evidence:
- evidence.md is append-only after Phase 1. Phase 4 probe results append to §Probe Log; nothing rewrites earlier sections.
Phase 2: Hypothesis Generation
Prospective gate (fires BEFORE spawning):
Coordinator outputs:
Cycle {N}: about to spawn 6 hypothesis agents + 1 outside-frame agent (Sonnet tier).
Estimated cost: ~${estimate} (cumulative: ${running_total} of ${budget_cap})
Dimensions queued: {list of angles' dimensions}
Required categories covered so far: {counts from state.json}
Continue? [y/N/redirect:<focus>]
Skip if --auto. If --auto AND cumulative budget exceeds 80% of cap: force user confirmation even in auto mode (budget backstop).
Step 2a — Enumerate angles (see DIMENSIONS.md):
For cycle 1:
- Generate 2–4 angles per dimension (8 dimensions × 2–4 = 16–32 angles)
- Generate 2–3 cross-dimensional angles
- Add
premortemangles from Phase 0e - Cap total frontier at 30. Priority-order per DIMENSIONS.md §Priority Ordering.
For cycle 2+:
- Start from graveyard (hypotheses rejected in prior cycle) — what dimensions were under-investigated?
- Add CRITICAL-priority angles for any dimension that had zero hypotheses accepted by the judge
- Add CRITICAL-priority angles for any required category still uncovered
Step 2b — Write data files before spawning:
deep-debug-{run_id}/known-hypotheses.md— list of all prior-cycle hypothesis IDs and 1-line titles (so critics don't re-propose)deep-debug-{run_id}/angles/{angle_id}.md— one file per angle with dimension, question, parent, rationaledeep-debug-{run_id}/evidence.mdis referenced by path; critics read it directly
Step 2c — Spawn hypothesis agents (parallel):
- Pop up to 6 highest-priority angles; assign to spec-derived hypothesis agents
- Spawn 1 additional outside-frame hypothesis agent (slot #7) — seeded from the symptom only, not the current hypothesis list or evidence file narrative
- Before each spawn: write
status: "in_progress",spawn_time_isoto state.json - Spawn each Agent in parallel (subagent_type: general-purpose, model: sonnet) with file paths — NOT inline content
- Hypothesis output path:
deep-debug-{run_id}/hypotheses/{angle_id}.md(content-addressed — coordinator CANNOT overwrite) - If Agent tool returns error: record
status: "spawn_failed", do not record as spawned; retry on next round
Quorum: Round complete if ≥ 4 of 6 spec-derived agents return parseable output within timeout (outside-frame tracked separately — its exit does not affect quorum).
Timeout scaling: 180s base. ×1.5 for cycles 2+ (larger evidence file).
Circuit breaker: ≥ 3 consecutive rounds with any critic failures → halt, log SYSTEM_FAILURE_ROUND, notify at turn boundary.
Step 2d — After agents complete:
- For each completed agent: read hypothesis file, extract hypothesis struct, dedup (see STATE.md Tier 1 + Tier 2), update state.json
- Spec-derived agents may suggest ≤ 1 new angle per cycle (immutable once written)
- Outside-frame agent may suggest ≤ 2 new angles (its exemption — it brings novel dimensions)
- Update coverage tracking: which dimensions / required categories are now covered?
Phase 3: Independent Hypothesis Judge (two-pass blind)
Purpose: Classify every hypothesis into plausibility tiers (leading | plausible | disputed | rejected | deferred) using a judge that cannot see the critic's confidence claim during pass 1. This prevents inflation.
Step 3a — Strip confidence from each hypothesis file:
- Coordinator produces
deep-debug-{run_id}/judge-inputs/batch_{cycle}_{batch_num}.md(up to 5 hypotheses per batch) - Stripping is mechanical: remove the
**Confidence:**line and theCONFIDENCE:field from STRUCTURED_OUTPUT - Stripping recorded in state.json as
judge_input_stripped: true - Original hypothesis files stay immutable
Step 3a-coherence — Cross-hypothesis coherence integrator:
Per _shared/cross-finding-coherence.md, adapted for hypotheses:
- Collect all parseable hypothesis output files from this cycle (post-dedup).
- Write integrator input manifest to
deep-debug-{run_id}/coherence/cycle-{N}-input.md. - Spawn Sonnet coherence-integrator agent. Output:
deep-debug-{run_id}/coherence/cycle-{N}-coherence.md. Timeout: 120s. - Parse structured annotations. For hypotheses, the key relationships are:
CONTRADICTS— two hypotheses make incompatible causal claims about the same component. At least one is wrong; the judge should scrutinize both.PATTERN_MEMBER— multiple hypotheses from different dimensions point to the same underlying architectural issue. This is early signal for Phase 7 escalation.SUPERSEDED_BY— one hypothesis is a strict subset of another (same mechanism, narrower scope). The judge should prefer the broader hypothesis.
- Attach annotations to
hypotheses.{id}.coherence_annotationin state.json. - Feed
GAPlines into the frontier as CRITICAL-priority angles for the next cycle. - If unparseable/timed out: proceed in degraded mode. Log
COHERENCE_DEGRADED.
Escalation signal: If the integrator identifies ≥ 3 hypotheses across different dimensions as PATTERN_MEMBER of the same root cause, this is structural evidence that the bug is architectural. Log COHERENCE_ARCHITECTURAL_SIGNAL — this does not trigger Phase 7 directly (fix_attempt_count is still the trigger) but the signal is surfaced in Phase 8 report.
Step 3b — ▶ Spawn three-model judge panel per batch (ENSEMBLE):
- Batch ≤ 5 hypotheses per agent (same as baseline)
- For EACH batch, spawn THREE independent judges in parallel, all with
run_in_background=true:judge_claude— Task tool, general-purpose subagent running Sonnet 4.6. Prompt: the pass-1 blind judge template from FORMAT.md plus instruction to write output todeep-debug-{run_id}/judge-verdicts/batch_{cycle}_{batch_num}_pass1_judge_claude.md.judge_openai— Task subagent that reads the batch input file, inlines it into the pass-1 judge prompt, callsmcp__pal__chat({model: "gpt-5.4", prompt: <full judge prompt with batch inlined>, working_directory_absolute_path: "/root"}), writes response VERBATIM to...pass1_judge_openai.md. See deep-qa-ensemble-v1 SKILL.md §"Canonical pal-subagent prompt" for the exact wrapper template — same pattern here.judge_gemini— identical tojudge_openaibut withmodel: "gemini"and output to...pass1_judge_gemini.md.
- Record batch in
background_tasks.judgeswithpass: 1and the extendedjudges[]sub-array capturing{id, model, provider, task_id, status, output_path}per spawn (same schema extension as deep-qa-ensemble-v1). - Timeout per judge: 90s (baseline); panel timeout 120s (overall).
- Watchdog: per-judge Monitor-based staleness kill at
STALE=3 min, HUNG=10 min. If ≥2 judges fire, batch →partial; aggregation handles it per the partial-results branch.
Step 3c — Two-pass judge protocol (see FORMAT.md §Judge Verdict):
- Pass 1: Judge classifies each hypothesis blind (no critic confidence claim). Applies 5 validation checks: falsifiability / contradiction / premise / evidence-grounding / simplicity (see §Validation Checks below). Writes pass-1 verdict to
deep-debug-{run_id}/judge-verdicts/batch_{cycle}_{batch_num}.md. - Pass 2: Coordinator provides the critic's confidence claim. Judge can CONFIRM, UPGRADE, or DOWNGRADE with rationale. Writes pass-2 addendum to the same file.
- Final plausibility is the pass-2 verdict.
Step 3d — Judge integrity check:
- After each cycle: if judge's
leading-rate ≥ 80% on ≥ 5 hypotheses, flagjudge_suspect: truein state and tag the cycle's leader[JUDGE_SUSPECT]. The coordinator does not re-decide — it surfaces the concern.
Step 3e — ▶ Drain ensemble panel + aggregate before Phase 4:
- For each batch in
background_tasks.judges, iterate thejudges[]sub-array and callTaskOutputwithblock=trueon each task_id. - A batch is drained when all three per-judge spawns have completed or timed out.
- Aggregate by invoking the deterministic aggregator via Bash:
python3 ~/.claude/skills/deep-debug-ensemble-v1/aggregate_ensemble_judges_debug.py \ --batch-id {batch_id} \ --judge-file judge_claude=deep-debug-{run_id}/judge-verdicts/batch_{cycle}_{batch_num}_pass1_judge_claude.md \ --judge-file judge_openai=deep-debug-{run_id}/judge-verdicts/batch_{cycle}_{batch_num}_pass1_judge_openai.md \ --judge-file judge_gemini=deep-debug-{run_id}/judge-verdicts/batch_{cycle}_{batch_num}_pass1_judge_gemini.md \ --out-json deep-debug-{run_id}/judge-verdicts/batch_{cycle}_{batch_num}_aggregated.json \ --out-summary-md deep-debug-{run_id}/judge-verdicts/batch_{cycle}_{batch_num}_aggregated.md - Read the aggregator's JSON output. For each hypothesis: apply aggregated
plausibility,evidence_tier,pass2_majority, andagreement_ratetohypotheses.{id}in state.json. Preserveper_model[]for post-hoc analysis. - Coordinator is NOT permitted to recompute plausibility from the per-judge files directly. The aggregator's output is authoritative.
- If a batch was flagged
failed(all 3 judges unparseable/timed out): retain critic-proposed plausibility, setjudge_status: "timed_out", logJUDGE_PANEL_TOTAL_FAILURE: batch_{id}. Partial panels (1-2 judges parseable) proceed normally — the aggregator's partial-results branch handles them.
Step 3f — Rebuttal round (if ≥ 2 hypotheses at leading or plausible):
- Spawn independent rebuttal agent (Sonnet) — separate from judge
- Input: leader + strongest alternative + evidence file
- Output:
deep-debug-{run_id}/rebuttal-cycle{N}.md - See EVIDENCE.md §Rebuttal Round for protocol
- Update plausibility based on rebuttal outcome
Phase 4: Discriminating Probe Execution
Purpose: Collapse uncertainty between the top-2 hypotheses with a concrete experiment. Not "think harder" — a command that produces hard evidence.
Step 4a — Probe design:
- If exactly 1 hypothesis at
leadingAND 0 hypotheses atplausible→ no probe needed; advance to Phase 5 with the leader - If ≥ 2 hypotheses at
leadingORplausible→ design a probe that distinguishes them
Probe requirements (see EVIDENCE.md §Discriminating Probe):
- Bounded time (≤30 minutes; ideally ≤10)
- Hard evidence output (log line, return value, test result)
- Pre-declared expected result for each hypothesis (post-hoc rationalization destroys probes)
- Reversible or non-destructive (touching production data requires user confirmation even in
--auto)
Step 4b — Write probe spec:
deep-debug-{run_id}/probes/probe_{id}.md(see FORMAT.md §Discriminating Probe Specification)probe_count_this_cycle += 1(hard cap: 3 per cycle)
Step 4c — Safety check:
- Does the probe write to production, delete data, or call an irreversible API? → USER CONFIRMATION REQUIRED
- If
--autoAND probe is destructive → reject the probe; choose a safer alternative or defer the hypothesis
Step 4d — Execute:
- Simple probe (test, query, check): coordinator runs directly
- Complex probe (build artifact, multi-step instrumentation): spawn
evidence-gathereragent (Haiku, background) with probe spec as input - Record raw output verbatim in
evidence.md§Probe Log (NO summarization)
Step 4e — Apply verdict:
- Match actual result against pre-declared expectations:
- Matches "expected if A" → A wins; B's status →
falsified_by_probe - Matches "expected if B" → B wins; A's status →
falsified_by_probe - Matches neither / matches both → inconclusive → design second probe (up to cap)
- Contradicts both → both to
rejected; cycle ends ashypothesis_space_saturated; re-enter Phase 2 next cycle with updated evidence
- Matches "expected if A" → A wins; B's status →
Step 4f — Winner promotion:
- If a winner emerges → its status becomes
promoted_to_fix; advance to Phase 5 - If probe cap hit without winner → cycle ends as
hypothesis_space_saturated;fix_attempt_count += 1(counts toward hard stop); advance to Phase 6
Phase 5: Fix + Verify
Purpose: Convert the promoted-to-fix hypothesis into working code with test-first discipline and verified pass.
REQUIRED sub-skills (invoke in order):
superpowers:test-driven-development— for writing the failing testsuperpowers:verification-before-completion— for confirming the fix
Step 5a — Write failing test (MANDATORY):
- Test must reproduce the exact symptom from Phase 0
- Run it — it MUST fail before the fix
- If no test framework exists → write a one-off reproduction script that exits non-zero on the bug
fix_attempts.{N}.failing_test_written: true→ update state.json- If you skip this step, the skill's Iron Law is violated — halt, require explicit
--allow-untestedoverride from user (rarely appropriate)
Step 5b — Implement ONE change:
- Address the hypothesis's identified mechanism
- ONE focused change; no bundling, no "while I'm here" refactoring
- Use
Edittool with explicit old/new strings - Write the diff to
deep-debug-{run_id}/fixes/fix_{N}.diff(usegit diff)
Step 5c — Verification:
- Re-run the failing test → MUST pass
- Run full test suite → MUST be clean (no new regressions)
- If any regression →
outcome: "failed", revert the change, record in state.json, advance to Phase 6 - Capture both test outputs into
evidence.md§Fix Attempts Log
Step 5d — Defense-in-depth (optional, strongly recommended):
- After a verified fix, add validation at layer boundaries per TECHNIQUES.md §Defense-in-Depth
- This is optional for cycle termination but is a report deliverable — the final report's Defense-in-Depth Suggestions section lists which layers the fix SHOULD get
Step 5e — Record outcome:
verified(test passes + suite clean)partial(test passes, suite reveals 1 small regression that's clearly related and easy to fix inline → fix it and re-verify)failed(test still fails, or suite has unrelated regressions)reverted(change had to be rolled back)
Phase 6: Cycle Termination Check
After Phase 5 outcome recorded:
| Outcome | Action |
|---|---|
verified |
Advance to Phase 8 with termination label Fixed — reproducing test now passes |
partial (re-verified inline) |
Same as verified |
failed AND fix_attempt_count < 3 |
Return to Phase 2 next cycle. Critical: the failure is NEW EVIDENCE — record what the fix changed, what didn't happen, what other failures appeared. Cycle 2's hypothesis agents get this in evidence.md |
failed AND fix_attempt_count >= 3 |
MANDATORY Phase 7 escalation. No Fix #4 under the current hypothesis set. |
reverted |
Same as failed — counts toward fix_attempt_count |
hypothesis_space_saturated (from Phase 3 or Phase 4) |
fix_attempt_count += 1 (counts even though no fix was attempted); if still < 3, return to Phase 2; if >= 3, Phase 7 |
Hard stop trigger: cycle >= hard_stop (default 6) → terminate with label Hard stop at cycle {N}; produce a partial report documenting the hypothesis graveyard.
Phase 7: Architectural Escalation
MANDATORY when fix_attempt_count >= 3 or hypothesis_space_saturated persists for 2 cycles.
Step 7a — Spawn architect agent (Opus — the one Opus spawn in this skill):
- Input: all prior cycles' hypothesis graveyards, all fix_attempts with outcomes, current evidence file, symptom
- Output:
deep-debug-{run_id}/architectural-question.mdper FORMAT.md §Architectural Question - Task: "Is this a pattern problem (wrong shared-state model), an invariant problem (assumed-but-violated precondition), or a wrong-abstraction problem (the component being debugged shouldn't exist in this shape)?"
Step 7b — Return to user:
- Print the architectural question to the user at turn boundary
- DO NOT attempt Fix #4 under any hypothesis from the prior 3 attempts
- To continue, the user must re-invoke deep-debug with
--hypothesis-override={chosen-architectural-alternative}after making the architectural decision
Step 7c — Terminate with label:
Architectural escalation required — 3 fix attempts failed across distinct hypotheses- Advance to Phase 8 with the architectural question as the primary deliverable
Phase 8: Final Report
- Do NOT read raw hypothesis files — use coordinator summary + judge verdicts + state.json
- Spawn Sonnet subagent to write
deep-debug-{run_id}/debug-report.md(see FORMAT.md §Final Debug Report) - Termination label must be from the 7-label vocabulary (below) — never improvised
- If termination is
Fixed: include resolved-cause section + defense-in-depth suggestions - If termination is
Environmental: include retry/monitoring contract - If termination is
Architectural escalation required: include reference toarchitectural-question.md - Coverage caveats MUST be prominent when required categories are uncovered
- If coherence integrator identified emergent patterns: include 'Cross-Dimensional Patterns' section showing which hypotheses shared root causes and whether the pattern signal aligned with the final fix hypothesis
Honest Termination Vocabulary (7 labels)
Exactly one applies at Phase 8:
| Label | When |
|---|---|
Fixed — reproducing test now passes |
Phase 5 verified: failing test passes + suite clean |
Environmental — requires retry/monitoring, not fix |
Investigation confirmed bug is not in code (flaky infrastructure, upstream service, timing dep) AND a retry/monitoring contract is specified |
Architectural escalation required — 3 fix attempts failed across distinct hypotheses |
Phase 7 triggered by fix_attempt_count >= 3 |
Hypothesis space saturated — no plausible hypothesis survives judge |
All dimensions explored, every hypothesis judge-rejected or probe-falsified |
Cannot reproduce — investigation blocked |
Phase 0 reproduction failed and user couldn't provide steps |
User-stopped at phase N |
User chose stop at a gate |
Hard stop at cycle N |
cycle >= hard_stop reached |
Never use a label not in this table. Never write "probably fixed" / "mostly fixed" / "should be fixed" / "no critical bugs remain."
Validation Checks
Every hypothesis that reaches the judge passes these 5 checks. Failing>disputed (not rejected). The judge applies them in pass 1.
Falsifiability check — Can you construct a concrete scenario where this hypothesis manifests AND a scenario where it does not? If not, the claim is unfalsifiable. Examples of unfalsifiable: "users might be slow sometimes" (no threshold), "there could be a race somewhere" (no specific ordering).
Contradiction check — Does this hypothesis contradict another accepted hypothesis or primary-tier evidence? If yes, at least one is misdiagnosed; flag both for re-examination.
Premise check — Would this hypothesis manifest in practice given the surrounding code/framework's existing guarantees? Examples that FAIL this check: "race between non-IO lines in gevent" (gevent cooperative scheduling prevents this), "SQLAlchemy cross-session mutation" (scoped sessions are isolated unless explicitly shared).
Evidence-grounding check — Does the hypothesis EXPLAIN the observed symptom, or is it consistent with it but not explanatory? Example: "bug is in file X" (consistent with stack trace but doesn't explain WHY the bug manifests) vs. "bug is in file X because
parse()returnsNonefor empty input, which the caller at Y doesn't check, causing AttributeError at Z" (explanatory).Simplicity check (Occam) — If two hypotheses explain the evidence equally, prefer the one with fewer assumptions. A hypothesis that requires "and also X is misconfigured and also Y is out of order" when a simpler hypothesis would explain the same evidence should be down-ranked.
Golden Rules
- No fixes without a hypothesis that survives independent judge + discriminating probe. The Iron Law. Violating the letter of this process is violating the spirit of debugging.
- Every hypothesis must be falsifiable with a concrete probe. "Could happen" is not a hypothesis; "would happen under X given Y, falsifiable by Z" is.
- Symptom-site ≠ cause-site — trace upstream. The stack trace points where the error surfaces, not where it originates. Use TECHNIQUES.md §Root Cause Tracing for the
data-flowdimension. - Evidence strength matters — rank, don't flatten. A tier-1 controlled reproduction outranks ten tier-5 circumstantial clues. See EVIDENCE.md §Evidence Strength Hierarchy.
- One fix at a time. No bundling, no "while I'm here" refactoring. If fix doesn't work, can't isolate what changed.
- Test-first for fixes. Invoke
superpowers:test-driven-development. The failing test is evidence you understand the bug; a passing test after the change is evidence you fixed the right thing. - Three failed fixes = architectural escalation (MANDATORY). No Fix #4 under the current hypothesis set. Phase 7 is not optional.
- Coordinator orchestrates; never self-classifies. Hypothesis plausibility, evidence tier, rebuttal verdict, probe winner — all delegated to independent agents. Coordinator reads structured output, doesn't produce verdicts.
- Framework guarantees are evidence — but verify, don't assume. "The ORM handles that" / "Gevent prevents that" / "The scoped session isolates that" — cite the specific code or docs that guarantee it, or the hypothesis relying on that guarantee is tier-4 inference, not tier-2 primary.
- Environmental bugs get explicit retry/monitoring contracts — never silent "it'll work next time." The
Environmentaltermination label requires a specific retry policy, monitoring, and escalation threshold. - Disputed hypotheses are documented, never silently dropped. The final report surfaces every hypothesis the judge rejected or disputed, with rationale.
- Judges must be adversarial. 100%
leading-rate across a cycle = broken judge (judge_suspect: true). Expected acceptance is 30–60% of critic-proposed hypotheses. - Disconfirmation must be attempted. A hypothesis without at least one disconfirmation search is
plausibleat best, neverleading. Confirmation-only evidence is confirmation bias dressed as investigation.
Anti-Rationalization Counter-Table
These are excuses the coordinator will be tempted to reach for under pressure. Each row is a defensive entry — when you catch yourself thinking the excuse, look at the reality.
| Excuse | Reality |
|---|---|
| "The stack trace points here — this is obviously the bug." | Symptom-site bias. The stack trace shows where the error surfaces, not where the data went wrong. Run root-cause-tracing (Golden Rule 3); the cause is often 3–5 frames upstream. |
| "I know this codebase — I don't need a hypothesis judge." | Overconfidence is the dominant mode of failed debugging. The judge's independence invariant applies whether or not you "know" the code — even experts anchor on the first fit. |
| "One more fix attempt, the pattern is almost there." | At 3+ failed fixes you're debugging the wrong abstraction. Phase 7 escalation is mandatory (Golden Rule 7). No Fix #4. |
| "This race condition is unlikely under normal load." | Unlikely × scale = certain. Apply the premise check: can you construct the scenario where it manifests? If yes, it's a real hypothesis — probe it, don't dismiss it. |
| "The test is flaky, not the code." | Sometimes. But "flaky" usually means race, ordering, or pollution. Invoke flaky-test-diagnoser before terminating as environmental. |
| "The framework prevents this (gevent / session scope / ORM lazy-loading)." | Framework guarantees are evidence — verify them in code, not in your head. Cite file:line of the guarantee. Un-cited framework claims are tier-4 inference, not tier-2 primary. (Golden Rule 9) |
| "Skip the failing test — I'll manually verify." | Manual verification is not a test. Untested fixes don't stick (the change surface gets edited, the bug comes back). Test-first is Golden Rule 6 for a reason. |
| "Multiple fixes at once saves time." | Can't isolate what worked. If symptom returns or a new bug appears, you have no signal on which change caused it. One fix at a time (Golden Rule 5). |
| "The probe would take 20 minutes — just fix it." | Probe cost is bounded (≤30 min). Fix-then-revert cost is unbounded (writing the fix, debugging its own issues, backing it out, re-hypothesizing). Run the probe. |
| "Judge downgraded this but my evidence is strong." | Judge saw the same evidence. If you disagree, use the challenge-token mechanism with a rationale — don't overrule silently. A coordinator that routinely overrides the judge has violated the independence invariant. |
| "It reproduced once — that's enough to commit." | One reproduction on your machine is not "fixed." Run the full test suite AND re-run the failing test multiple times (if concurrency-related). Only Fixed — reproducing test now passes requires both the targeted test AND the full suite. |
| "Previous hypothesis was close — just tweak it." | A changed hypothesis gets a fresh judge pass from scratch. Previous leading verdicts on pre-mutation hypotheses are stale. |
| "I don't need a discriminating probe; I'll look at the code more carefully." | Re-reading code is gathered evidence, not new evidence. Probes collapse uncertainty by producing hard observations (log line, test result, return value). Code review re-confirms what you already believed. (EVIDENCE.md §Discriminating Probe) |
| "The symptom and the recent change are obviously unrelated." | recent-changes is a required dimension precisely because "obviously unrelated" changes are how regressions hide. The judge will demand the linking probe anyway. |
| "This looks environmental — skip to retry." | The Environmental termination label requires completed investigation across all 8 dimensions AND a retry/monitoring contract. "Skip investigation" is not a label. |
| "I found ONE hypothesis that fits — that's the answer." | One hypothesis that fits evidence is one of several that MAY fit. Generate competitors (Phase 2) before accepting. Anchoring on the first fit is the second-most-common failure mode of solo debugging. |
| "The 5 validation checks are overkill for this simple bug." | Simple bugs pass the checks quickly — running them costs nothing. "Skip because simple" is the rationalization that lets unvalidated hypotheses become fixes that don't hold. |
| "I'll update the symptom to match the new understanding." | Symptom is locked at Phase 0 with sha256 verification. Silent rewrite = SYMPTOM_TAMPERED and halts the run. If new understanding changes the symptom, start a new run with a new symptom. |
| "Probes fired, no winner — I'll just pick the better-looking one." | Inconclusive probe ⇒ inconclusive. Either design another probe (up to cap) or accept hypothesis_space_saturated — don't silently pick a loser. |
| "3 failed fixes is bad luck — the next will hold." | 3 failed fixes is a structural signal, not bad luck. Each fix revealed new information; the pattern is consistent: wrong abstraction. Phase 7 escalates. |
When you reach for any of these rationalizations:
…(truncated)