Systematic Debugging
Invocation and Resource Paths
This module is source-only by default. A user request for this workflow is enough;
do not assume its slash command is registered. Resolve its exact SKILL.md from
the project, the current runtime's active skill root, then the matching
SKILLS-CATALOG.md row. module_root is the directory containing that file;
read supporting documents relative to it, regardless of the project working directory.
Harness Integration
Argos and Minos read this module when an observed failure enters their authorized repair
loop. They retain ownership of scope, issue IDs, reports, retry budgets, and final grades.
Reading this module does not start another loop, request TDD, or grant permission to edit.
Inspection-only requests remain read-only; a report-only command only displays the existing report.
- Reuse the recorded failing check when its code, input, and environment still match.
For an exploratory issue without a test, identify a repeatable observation or focused check first.
- Separate the current requirement/AC from the actual response or state. An error label is a clue,
not a root cause: a timeout may be a server failure rather than a reason to add a wait.
- Record one testable cause and the observation that supports or would disprove it before editing.
If existing evidence already establishes the cause, fix it directly without repeating the investigation.
- Make the smallest authorized change, then rerun the original check and affected regression checks.
A source-code edit alone or green tests that never covered the reported issue do not close it.
- If the attempt fails, preserve its evidence, restore only that attempt's known changes when needed,
and return to the unanswered question within the caller's remaining budget. Do not reset counts by
changing workers, sessions, issue labels, or entering this module. Repeated identical output without
new evidence is a blocker to report, not a reason for another speculative patch.
Put issue/AC, reproduction, contract vs actual, hypothesis/evidence, change, verification, remaining budget in the caller's existing repair history. Do not create a second report or ledger.
Missing prerequisites are NOT RUN/UNVERIFIED; an unresolved issue retains its actual severity.
Use the phases below only as far as the remaining uncertainty requires.
Overview
Random fixes waste time and create new bugs. Quick patches mask underlying issues.
Core principle: Investigate the cause before choosing a fix, and distinguish
evidence-backed diagnosis from temporary symptom mitigation.
The Iron Law
NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST
Gather enough evidence for a testable hypothesis before changing behavior.
During an incident, an authorized rollback or containment may precede full diagnosis;
record it as mitigation and preserve the evidence needed to investigate the cause.
The Four Phases
Use the phases to reduce uncertainty. Reuse evidence already established for this
code and environment; a clear local failure does not require a multi-component investigation.
Phase 1: Root Cause Investigation
BEFORE attempting ANY fix:
Read Error Messages Carefully
- Don't skip past errors or warnings
- They often contain the exact solution
- Read stack traces completely
- Note line numbers, file paths, error codes
Reproduce Consistently
- Can you trigger it reliably?
- What are the exact steps?
- Does it happen every time?
- If not reproducible → gather more data, don't guess
Check Recent Changes
- What changed that could cause this?
- Git diff, recent commits
- New dependencies, config changes
- Environmental differences
Gather Evidence in Multi-Component Systems
WHEN system has multiple components (CI → build → signing, API → service → database):
When existing evidence does not locate the failing boundary, add focused instrumentation:
For EACH component boundary:
- Record non-sensitive input/output shapes or redacted diagnostic values
- Verify environment/config propagation
- Check state at each layer
Run once to gather evidence showing WHERE it breaks
THEN analyze evidence to identify failing component
THEN investigate that specific component
Example (multi-layer system):
# Run the same presence check at the workflow and build-script boundaries.
# Report configuration presence without printing its value or the full environment.
if [ -n "${IDENTITY:-}" ]; then
echo "IDENTITY: SET"
else
echo "IDENTITY: UNSET"
fi
This reveals: Which layer fails (secrets → workflow ✓, workflow → build ✗)
Trace Data Flow
WHEN error is deep in call stack:
Read root-cause-tracing.md under module_root for the complete backward tracing technique.
Quick version:
- Where does bad value originate?
- What called this with bad value?
- Keep tracing up until you find the source
- Fix at source, not at symptom
Phase 2: Pattern Analysis
Find the pattern before fixing:
Find Working Examples
- Locate similar working code in same codebase
- What works that's similar to what's broken?
Compare Against References
- Read the reference implementation and callers needed to understand the affected contract.
- Expand the search only when a concrete dependency or unanswered question requires it.
Identify Differences
- What's different between working and broken?
- List every difference, however small
- Don't assume "that can't matter"
Understand Dependencies
- What other components does this need?
- What settings, config, environment?
- What assumptions does it make?
Phase 3: Hypothesis and Testing
Scientific method:
Form Single Hypothesis
- State clearly: "I think X is the root cause because Y"
- Write it down
- Be specific, not vague
Test Minimally
- Make the SMALLEST possible change to test hypothesis
- One variable at a time
- Don't fix multiple things at once
Verify Before Continuing
- Did it work? Yes → Phase 4
- Didn't work? Form NEW hypothesis
- DON'T add more fixes on top
When You Don't Know
- Say "I don't understand X"
- Don't pretend to know
- Ask for help
- Research more
Phase 4: Implementation
Fix the root cause, not the symptom:
Create Failing Test Case
- Simplest possible reproduction
- Automated test if possible
- One-off test script if no framework
- MUST have before fixing
- Use the project's existing test framework or a reproducible check for the original symptom.
- Requesting this debugging workflow does not also request strict TDD. Only if the user explicitly
requests TDD, resolve
test-driven-development by the same project → active root → exact catalog
path order above and read its SKILL.md directly. If unavailable, report that guidance as
NOT RUN and continue the authorized reproduction and verification with project-native tools.
Implement Single Fix
- Address the root cause identified
- ONE change at a time
- No "while I'm here" improvements
- No bundled refactoring
Verify Fix
- Run the original reproduction against the changed code and verify the expected response,
persisted state, and relevant side effects; a changed file is not proof that the symptom is fixed.
- Run the project's required tests/build plus regression checks for the affected behavior.
Record the command, target revision or working-tree state, environment, exit code, and result.
- Reuse an existing result only when code, environment, and checked conditions still match;
rerun affected checks after a change or when a specific unresolved doubt requires it.
- Report
NOT RUN with the missing prerequisite when execution is unavailable. Distinguish
passing checks, failing checks, and unverified requirements before claiming completion.
If Fix Doesn't Work
- Record the hypothesis, change, and observed result before choosing another experiment.
- Restore only this experiment's changes when the baseline is known; preserve existing user
changes and other work. Do not reset files or discard a branch to restart the process.
- Return to the earliest unanswered question. A missing prerequisite, wrong assumption,
implementation error, test defect, and architecture problem need different responses.
- Repeated failure warrants reassessment; the number of attempts alone does not diagnose
the architecture. Shared-state coupling or incompatible contracts must be demonstrated.
- Follow the current task's retry/time budget. Continue an authorized, evidence-backed fix;
report the blocker and next discriminating check if further attempts only repeat guesses.
Discuss scope-changing architectural work before undertaking it.
Red Flags - STOP and Follow Process
If you catch yourself thinking:
- "Quick fix for now, investigate later"
- "Just try changing X and see if it works"
- "Add multiple changes, run tests"
- "Skip the test, I'll manually verify"
- "It's probably X, let me fix that"
- "I don't fully understand but this might work"
- "Pattern says X but I'll adapt it differently"
- "Here are the main problems: [lists fixes without investigation]"
- Proposing solutions before tracing data flow
- "One more fix attempt" without a new observation or testable hypothesis
- Each fix reveals new problem in different place
ALL of these mean: STOP. Return to Phase 1.
your human partner's Signals You're Doing It Wrong
Watch for these redirections:
- "Is that not happening?" - You assumed without verifying
- "Will it show us...?" - You should have added evidence gathering
- "Stop guessing" - You're proposing fixes without understanding
- "Ultrathink this" - Question fundamentals, not just symptoms
- "We're stuck?" (frustrated) - Your approach isn't working
When you see these: STOP. Return to Phase 1.
Quick Reference
| Phase |
Key Activities |
Success Criteria |
| 1. Root Cause |
Read errors, reproduce, check changes, gather evidence |
Understand WHAT and WHY |
| 2. Pattern |
Find working examples, compare |
Identify differences |
| 3. Hypothesis |
Form theory, test minimally |
Confirmed or new hypothesis |
| 4. Implementation |
Create test, fix, verify |
Bug resolved, tests pass |
When Process Reveals "No Root Cause"
If systematic investigation reveals issue is truly environmental, timing-dependent, or external:
- You've completed the process
- Document what you investigated
- Implement appropriate handling (retry, timeout, error message)
- Add monitoring/logging for future investigation
State the unresolved alternatives and evidence limits. An exhausted investigation is
not proof of an external cause, and a successful mitigation is not proof of a root-cause fix.
Supporting Techniques
These techniques are part of systematic debugging and available in this directory:
- root-cause-tracing.md - Trace bugs backward through call stack to find original trigger
- defense-in-depth.md - Add validation at multiple layers after finding root cause
- condition-based-waiting.md - Replace arbitrary timeouts with condition polling
Strict TDD is an optional, explicitly requested source module (Phase 4, Step 1).
Completion evidence is owned by Phase 4, Step 3 above; no external verification skill is required.
1---2name: systematic-debugging3description: |4 Structured root-cause investigation when explicitly requested, or as an internal5 reference in Argos/Minos authorized repair loops. Routine diagnosis stays native.6metadata:7 source: https://skillmd.com/skills/dannykkh/systematic-debugging8---910# Systematic Debugging1112## Invocation and Resource Paths1314This module is source-only by default. A user request for this workflow is enough;15do not assume its slash command is registered. Resolve its exact `SKILL.md` from16the project, the current runtime's active skill root, then the matching17`SKILLS-CATALOG.md` row. `module_root` is the directory containing that file;18read supporting documents relative to it, regardless of the project working directory.1920## Harness Integration2122Argos and Minos read this module when an observed failure enters their authorized repair23loop. They retain ownership of scope, issue IDs, reports, retry budgets, and final grades.24Reading this module does not start another loop, request TDD, or grant permission to edit.25Inspection-only requests remain read-only; a report-only command only displays the existing report.2627- Reuse the recorded failing check when its code, input, and environment still match.28 For an exploratory issue without a test, identify a repeatable observation or focused check first.29- Separate the current requirement/AC from the actual response or state. An error label is a clue,30 not a root cause: a timeout may be a server failure rather than a reason to add a wait.31- Record one testable cause and the observation that supports or would disprove it before editing.32 If existing evidence already establishes the cause, fix it directly without repeating the investigation.33- Make the smallest authorized change, then rerun the original check and affected regression checks.34 A source-code edit alone or green tests that never covered the reported issue do not close it.35- If the attempt fails, preserve its evidence, restore only that attempt's known changes when needed,36 and return to the unanswered question within the caller's remaining budget. Do not reset counts by37 changing workers, sessions, issue labels, or entering this module. Repeated identical output without38 new evidence is a blocker to report, not a reason for another speculative patch.3940Put `issue/AC, reproduction, contract vs actual, hypothesis/evidence, change, verification,41remaining budget` in the caller's existing repair history. Do not create a second report or ledger.42Missing prerequisites are `NOT RUN`/`UNVERIFIED`; an unresolved issue retains its actual severity.43Use the phases below only as far as the remaining uncertainty requires.4445## Overview4647Random fixes waste time and create new bugs. Quick patches mask underlying issues.4849**Core principle:** Investigate the cause before choosing a fix, and distinguish50evidence-backed diagnosis from temporary symptom mitigation.5152## The Iron Law5354```55NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST56```5758Gather enough evidence for a testable hypothesis before changing behavior.59During an incident, an authorized rollback or containment may precede full diagnosis;60record it as mitigation and preserve the evidence needed to investigate the cause.6162## The Four Phases6364Use the phases to reduce uncertainty. Reuse evidence already established for this65code and environment; a clear local failure does not require a multi-component investigation.6667### Phase 1: Root Cause Investigation6869**BEFORE attempting ANY fix:**70711. **Read Error Messages Carefully**72 - Don't skip past errors or warnings73 - They often contain the exact solution74 - Read stack traces completely75 - Note line numbers, file paths, error codes76772. **Reproduce Consistently**78 - Can you trigger it reliably?79 - What are the exact steps?80 - Does it happen every time?81 - If not reproducible → gather more data, don't guess82833. **Check Recent Changes**84 - What changed that could cause this?85 - Git diff, recent commits86 - New dependencies, config changes87 - Environmental differences88894. **Gather Evidence in Multi-Component Systems**9091 **WHEN system has multiple components (CI → build → signing, API → service → database):**9293 **When existing evidence does not locate the failing boundary, add focused instrumentation:**94 ```95 For EACH component boundary:96 - Record non-sensitive input/output shapes or redacted diagnostic values97 - Verify environment/config propagation98 - Check state at each layer99100 Run once to gather evidence showing WHERE it breaks101 THEN analyze evidence to identify failing component102 THEN investigate that specific component103 ```104105 **Example (multi-layer system):**106 ```bash107 # Run the same presence check at the workflow and build-script boundaries.108 # Report configuration presence without printing its value or the full environment.109 if [ -n "${IDENTITY:-}" ]; then110 echo "IDENTITY: SET"111 else112 echo "IDENTITY: UNSET"113 fi114 ```115116 **This reveals:** Which layer fails (secrets → workflow ✓, workflow → build ✗)1171185. **Trace Data Flow**119120 **WHEN error is deep in call stack:**121122 Read [root-cause-tracing.md](root-cause-tracing.md) under `module_root` for the complete backward tracing technique.123124 **Quick version:**125 - Where does bad value originate?126 - What called this with bad value?127 - Keep tracing up until you find the source128 - Fix at source, not at symptom129130### Phase 2: Pattern Analysis131132**Find the pattern before fixing:**1331341. **Find Working Examples**135 - Locate similar working code in same codebase136 - What works that's similar to what's broken?1371382. **Compare Against References**139 - Read the reference implementation and callers needed to understand the affected contract.140 - Expand the search only when a concrete dependency or unanswered question requires it.1411423. **Identify Differences**143 - What's different between working and broken?144 - List every difference, however small145 - Don't assume "that can't matter"1461474. **Understand Dependencies**148 - What other components does this need?149 - What settings, config, environment?150 - What assumptions does it make?151152### Phase 3: Hypothesis and Testing153154**Scientific method:**1551561. **Form Single Hypothesis**157 - State clearly: "I think X is the root cause because Y"158 - Write it down159 - Be specific, not vague1601612. **Test Minimally**162 - Make the SMALLEST possible change to test hypothesis163 - One variable at a time164 - Don't fix multiple things at once1651663. **Verify Before Continuing**167 - Did it work? Yes → Phase 4168 - Didn't work? Form NEW hypothesis169 - DON'T add more fixes on top1701714. **When You Don't Know**172 - Say "I don't understand X"173 - Don't pretend to know174 - Ask for help175 - Research more176177### Phase 4: Implementation178179**Fix the root cause, not the symptom:**1801811. **Create Failing Test Case**182 - Simplest possible reproduction183 - Automated test if possible184 - One-off test script if no framework185 - MUST have before fixing186 - Use the project's existing test framework or a reproducible check for the original symptom.187 - Requesting this debugging workflow does not also request strict TDD. Only if the user explicitly188 requests TDD, resolve `test-driven-development` by the same project → active root → exact catalog189 path order above and read its `SKILL.md` directly. If unavailable, report that guidance as190 `NOT RUN` and continue the authorized reproduction and verification with project-native tools.1911922. **Implement Single Fix**193 - Address the root cause identified194 - ONE change at a time195 - No "while I'm here" improvements196 - No bundled refactoring1971983. **Verify Fix**199 - Run the original reproduction against the changed code and verify the expected response,200 persisted state, and relevant side effects; a changed file is not proof that the symptom is fixed.201 - Run the project's required tests/build plus regression checks for the affected behavior.202 Record the command, target revision or working-tree state, environment, exit code, and result.203 - Reuse an existing result only when code, environment, and checked conditions still match;204 rerun affected checks after a change or when a specific unresolved doubt requires it.205 - Report `NOT RUN` with the missing prerequisite when execution is unavailable. Distinguish206 passing checks, failing checks, and unverified requirements before claiming completion.2072084. **If Fix Doesn't Work**209 - Record the hypothesis, change, and observed result before choosing another experiment.210 - Restore only this experiment's changes when the baseline is known; preserve existing user211 changes and other work. Do not reset files or discard a branch to restart the process.212 - Return to the earliest unanswered question. A missing prerequisite, wrong assumption,213 implementation error, test defect, and architecture problem need different responses.214 - Repeated failure warrants reassessment; the number of attempts alone does not diagnose215 the architecture. Shared-state coupling or incompatible contracts must be demonstrated.216 - Follow the current task's retry/time budget. Continue an authorized, evidence-backed fix;217 report the blocker and next discriminating check if further attempts only repeat guesses.218 Discuss scope-changing architectural work before undertaking it.219220## Red Flags - STOP and Follow Process221222If you catch yourself thinking:223- "Quick fix for now, investigate later"224- "Just try changing X and see if it works"225- "Add multiple changes, run tests"226- "Skip the test, I'll manually verify"227- "It's probably X, let me fix that"228- "I don't fully understand but this might work"229- "Pattern says X but I'll adapt it differently"230- "Here are the main problems: [lists fixes without investigation]"231- Proposing solutions before tracing data flow232- **"One more fix attempt" without a new observation or testable hypothesis**233- **Each fix reveals new problem in different place**234235**ALL of these mean: STOP. Return to Phase 1.**236237## your human partner's Signals You're Doing It Wrong238239**Watch for these redirections:**240- "Is that not happening?" - You assumed without verifying241- "Will it show us...?" - You should have added evidence gathering242- "Stop guessing" - You're proposing fixes without understanding243- "Ultrathink this" - Question fundamentals, not just symptoms244- "We're stuck?" (frustrated) - Your approach isn't working245246**When you see these:** STOP. Return to Phase 1.247248## Quick Reference249250| Phase | Key Activities | Success Criteria |251|-------|---------------|------------------|252| **1. Root Cause** | Read errors, reproduce, check changes, gather evidence | Understand WHAT and WHY |253| **2. Pattern** | Find working examples, compare | Identify differences |254| **3. Hypothesis** | Form theory, test minimally | Confirmed or new hypothesis |255| **4. Implementation** | Create test, fix, verify | Bug resolved, tests pass |256257## When Process Reveals "No Root Cause"258259If systematic investigation reveals issue is truly environmental, timing-dependent, or external:2602611. You've completed the process2622. Document what you investigated2633. Implement appropriate handling (retry, timeout, error message)2644. Add monitoring/logging for future investigation265266State the unresolved alternatives and evidence limits. An exhausted investigation is267not proof of an external cause, and a successful mitigation is not proof of a root-cause fix.268269## Supporting Techniques270271These techniques are part of systematic debugging and available in this directory:272273- [root-cause-tracing.md](root-cause-tracing.md) - Trace bugs backward through call stack to find original trigger274- [defense-in-depth.md](defense-in-depth.md) - Add validation at multiple layers after finding root cause275- [condition-based-waiting.md](condition-based-waiting.md) - Replace arbitrary timeouts with condition polling276277Strict TDD is an optional, explicitly requested source module (Phase 4, Step 1).278Completion evidence is owned by **Phase 4, Step 3** above; no external verification skill is required.279