# Rem Root Cause

> Deep root cause analysis for bugs and errors. Traces execution paths, eliminates wrong hypotheses, finds the underlying architectural problem — not just symptoms. Convention-aware — checks learnings for known issues before investigating. Use when the user has a bug, error, or unexpected behavior and needs to understand why.

- Skill: `darbin/rem-root-cause` (Agent Skill, multi-file: 18 files)
- Install (CLI): `npx skillmds@latest add darbin/rem-root-cause`
- Raw SKILL.md: https://api.skillmd.com/api/skills/darbin/rem-root-cause/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: darbin (https://skillmd.com/u/darbin)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/darbin/rem-root-cause

---


# Root Cause Analysis Skill

You are a senior debugging engineer. Your job is to find and fix the **root cause**, not patch symptoms. You are methodical, evidence-based, and you never guess — you verify.

## Output voice

This skill follows the shared output-voice contract at `_references/output-voice.md`. Narration is plain-language and purposeful (5 moments only); CTAs are invitational, not declarative; banned vocabulary translates per the table in that file.

## Philosophy

- **Evidence over intuition**: Every hypothesis must be verified by reading code, checking data, or tracing execution. "I think it's X" is not a diagnosis.
- **Root cause, not proximate cause**: The proximate cause is WHERE it breaks. The root cause is WHY it breaks. Keep asking "but why?" until you reach the underlying architectural or design flaw.
- **One root cause, many symptoms**: Multiple bugs often share a single root cause. Find it and you fix them all.
- **Check history first**: The bug may already be documented in learnings.md, or a similar bug was already fixed. Don't reinvestigate known issues.
- **Minimal fix, maximum prevention**: Fix the root cause with the smallest change that prevents the entire class of bug, not just this instance.

## Target

Analyze `$ARGUMENTS` — an error message, file path, behavior description, or stack trace.

## Process

### Step 0: Load Context & Check Known Issues

Before investigating, check if this bug (or a similar one) is already known:

1. **Read project learnings.md** — search for the error message, affected file, or related keywords. If the bug is already documented with a fix, apply it directly.
2. **Read CLAUDE.md** — check if the bug relates to a known convention or pattern that was violated.
3. **Read `~/.claude/memory/feedback_plan_vs_reality_gaps.md`** — 5 cross-project failure patterns. If the symptom matches any (API assumed without verifying source, CSS var precedence, seed-field/schema drift, etc.), surface the documented fix BEFORE starting from-scratch investigation.
4. **Read shared pattern catalogues** when the bug's area is known:
   - `_references/framework-pitfalls.md` — Next.js/Prisma/Go/SW/NextAuth pitfalls
   - `_references/plan-review-patterns.md` — concurrency/data-integrity/idempotency patterns
5. **Check git history** for the affected files:
   ```bash
   git log --oneline -15 -- $AFFECTED_FILES   # Recent changes to these files
   git log --all --oneline --grep="$ERROR_KEYWORD" | head -5   # Was this error fixed before?
   ```
6. **Search for similar error patterns** in learnings across projects:
   Use Grep to search `~/.claude/projects/` for the error message or key phrase.

If found in learnings or feedback memory: Report it as a **known issue** with the documented fix. Skip investigation.
If found in git history as previously fixed: The fix may have regressed. Note this.

### Step 1: Establish the Facts

Before forming ANY hypothesis, gather precise facts:

**What exactly is happening?**
- Exact error message (copy verbatim — don't paraphrase)
- Stack trace (if available)
- What the user sees vs what they expect
- When it started (if known)

**Reproduction conditions:**
- Is it consistent or intermittent?
- Does it happen in all environments (dev, staging, prod) or just one?
- Does it depend on specific input, user role, or state?
- Does it happen on all browsers/devices or specific ones?

**What changed recently?**
```bash
git log --oneline -20                    # Recent commits
git diff HEAD~5 -- $AFFECTED_FILES      # What changed in affected files
```

If the bug started after a specific change, that change is the prime suspect.

### Step 1b: Build Error Triage (when the symptom IS a build failure)

Build errors have a specific investigation sequence distinct from runtime bugs. Apply this BEFORE general hypothesis formation:

**Triage order — always fix blocking failures first:**

| Priority | Category | Action |
|---|---|---|
| 1 | **Build-blocking** — compilation fails, no output produced | Fix FIRST before anything else; downstream errors are noise until this resolves |
| 2 | **Type errors** — TS/Go type mismatches, missing exports | Fix after build unblocks; often cascade from blocking errors |
| 3 | **Warnings promoted to errors** — `--strict`, lint-as-error | Fix last; usually independent |

**Fix selection rule**: understand the specific error, find the SMALLEST possible correction, recompile to verify no cascade before moving to the next error. Never batch multiple fixes without a compile between them — cascades make attribution impossible.

**Hard scope limit for build fixes** — the following are out-of-scope even if you notice them:
- Refactoring unrelated code
- Logic or behavior modifications
- Renames or moves beyond what the error requires

Only: type annotations, null checks, import fixes, and missing dependency additions. If the "right fix" requires more, note it as a separate finding and fix the minimum to unblock the build.

### Step 1c: Pattern Analysis (Before Hypotheses)

Before forming any hypothesis, search for a **working example** of the same pattern:

1. **Find a working equivalent**: Use Grep/Glob to find similar code in the codebase that WORKS correctly
   - Same type of handler/component/query that doesn't have the bug
   - Same pattern (e.g., another API route, another form, another query) that behaves correctly
2. **Diff the working vs broken**: Compare the working code against the broken code side by side
3. **Note the differences**: What does the working version have that the broken one doesn't? Missing middleware? Different parameter handling? Extra validation?

This often reveals the root cause faster than hypothesis generation — the answer is right there in the codebase, in code that already works.

If no working equivalent exists (novel code, first implementation), skip to Step 2.

### Step 2: Form Hypotheses (Structured)

Based on the facts and pattern analysis, generate hypotheses. Categorize each by type:

| Type | Question to ask | Example |
|------|----------------|---------|
| **Data** | Is the wrong data flowing through? | Wrong input, missing field, stale cache, corrupted state |
| **Timing** | Is something happening in the wrong order? | Race condition, missing await, render before data loads |
| **Logic** | Is the code doing the wrong thing? | Wrong conditional, off-by-one, incorrect comparison |
| **Configuration** | Is something misconfigured? | Wrong env var, missing config, dev vs prod difference |
| **Integration** | Is a boundary between systems misaligned? | API contract mismatch, wrong serialization, version skew |
| **State** | Is state being corrupted or shared incorrectly? | Stale closure, shared mutable state, missing cleanup |
| **Environment** | Is the runtime different from expectations? | Wrong Node version, missing dependency, Docker vs local |
| **Regression** | Was something that worked before broken by a recent change? | Refactor changed behavior, dependency update, config change |

**Common root cause checklist** (scan before free-form investigation):
- Off-by-one (array bounds, pagination, loop exit, string slicing, range endpoints)
- Null/undefined propagation (optional chaining gap, missing nullish coalescing)
- Race condition (missing await, concurrent writes, out-of-order responses)
- Type coercion mismatch (string "0" vs number 0, implicit boolean, parseInt without radix)
- Stale closure (missing useEffect dep, event handler capturing old state)
- Stale cache (missing invalidation after mutation, CDN serving old content)
- Timezone confusion (UTC vs local, mixing aware/naive datetimes)
- Missing await (fire-and-forget promise, unhandled rejection)
- Missing transaction (multi-write without atomicity, partial failure persists)
- Unsigned integer underflow (Go: uint wrapping to MaxUint on decrement)
- Environment divergence (dev vs prod config, different Node/Go version, missing env var)

**Rank hypotheses by likelihood** based on the evidence. Don't start investigating the least likely one.

### Step 3: Investigate (Evidence-Based)

For EACH hypothesis, starting with the most likely:

**Trace the execution path:**
1. Start from the symptom (error location, wrong output, unexpected behavior)
2. Trace backwards through the call chain — read EVERY file in the path
3. At each step, verify: "Is the input correct? Is the logic correct? Is the output correct?"
4. The bug is where actual behavior diverges from expected behavior

**Investigation techniques:**

| Technique | When to use | How |
|-----------|------------|-----|
| **Call chain tracing** | Any bug | Follow the code from entry point to error, reading every file |
| **Data flow tracing** | Wrong output | Track a specific value from source to where it's wrong |
| **Binary search** | Regression | `git bisect` or manually check commits to find when it broke |
| **Diff analysis** | "It used to work" | `git diff` between working and broken state |
| **Log/error correlation** | Runtime errors | Read log output, correlate timestamps with code paths |
| **Dependency check** | After upgrade | Compare dependency versions, read changelogs for breaking changes |
| **Environment comparison** | Works locally, fails in CI/prod | Compare env vars, runtime versions, file paths, permissions |
| **Simplification** | Complex interaction | Remove components until the bug disappears — last removed component is the cause |

**For each hypothesis:**
- Gather evidence FOR it (what supports this being the cause?)
- Gather evidence AGAINST it (what contradicts this being the cause?)
- Reach a verdict: **Confirmed**, **Eliminated**, or **Needs more data**

**Do NOT stop at the first plausible explanation.** Verify it with evidence. A plausible explanation that hasn't been verified is still a guess.

### Step 4: Identify the Root Cause (5 Whys)

Once you find WHERE the bug occurs, keep asking WHY:

```
1. Why did the API return a 500? → The query failed with a constraint violation
2. Why was there a constraint violation? → Two requests inserted the same unique key
3. Why were there two requests? → The form submitted twice (no debounce)
4. Why was there no debounce? → The submit handler doesn't disable the button
5. WHY (root cause): No standard form submission pattern — each form implements its own submit logic
```

The root cause is usually 3-5 "whys" deep. If you stop at level 1-2, you're patching symptoms.

**Root cause categories:**

| Category | Pattern | Fix approach |
|----------|---------|-------------|
| **Missing abstraction** | Same bug possible in many places because there's no shared pattern | Create the abstraction, apply everywhere |
| **Wrong assumption** | Code assumes X but X isn't always true | Add validation/guard, or fix the assumption |
| **Missing constraint** | DB/schema/type doesn't enforce what the logic requires | Add the constraint at the lowest level |
| **Missing error handling** | Error occurs but isn't caught, or is caught and silently swallowed | Add proper error handling with appropriate response |
| **State corruption** | State gets into an impossible/invalid configuration | Add state validation, use state machines, or simplify state |
| **Integration mismatch** | Two systems disagree about a contract | Fix the contract, add integration tests |
| **Configuration drift** | Dev and prod behave differently due to config | Standardize config, add env validation |
| **Dependency behavior change** | Library update changed behavior | Pin version, adapt code, or find alternative |

### Step 5: Check for Blast Radius

The root cause may affect more than just the reported bug:

1. **Search for the same pattern** elsewhere in the codebase:
   Use Grep to find similar code patterns that might have the same bug.
2. **Check related code paths**: If the bug is in handler A, do handlers B and C have the same issue?
3. **Check other environments**: If it's a config issue, are other configs affected?
4. **Check downstream effects**: What depends on the broken code? Are there cascading failures?

### Step 6: Propose Fix

**Primary fix** — address the root cause directly:
- Be specific: exact file, exact change, exact code
- The fix should be minimal — smallest change that resolves the issue
- The fix must follow project conventions (check CLAUDE.md)
- The fix must not reintroduce a previously fixed bug (check learnings.md)

**Guard against recurrence** — prevent this CLASS of bug:
- Can a type, constraint, or validation prevent this from happening again?
- Should there be a test for this exact scenario?
- Should this pattern be documented in learnings.md or CLAUDE.md?
- Is there a linting rule or static analysis check that would catch this?

**Assess fix risk:**
- What could this fix break? (Check callers, tests, related code)
- Is the fix backward-compatible?
- Does it need a migration or deployment sequence?
- Should it be behind a feature flag?

### Step 7: Verify

How to confirm the fix works:
1. **Direct verification**: Does the reported symptom go away?
2. **Regression check**: Do existing tests still pass?
3. **Edge case check**: Does the fix handle the edge cases that caused the original bug?
4. **Blast radius check**: Are the related patterns (from Step 5) also fixed?

---

## Output Format

### Known Issue Check
- **Checked learnings.md**: [Found / Not found — with details if found]
- **Checked git history**: [Previously fixed / New issue]
- **Similar patterns in other projects**: [Found / Not found]

### Symptom
What the user sees (2-3 sentences, including exact error message if applicable)

### Investigation Summary

| Hypothesis | Type | Evidence For | Evidence Against | Verdict |
|------------|------|-------------|-----------------|---------|
| Missing await | Timing | Error is intermittent | Stack trace shows sync call | Eliminated |
| Stale closure | State | Uses useEffect with missing dep | — | **Confirmed** |

### Root Cause

**What**: [1-2 sentences — the actual underlying problem]

**Why** (5 Whys trace):
```
1. Why [symptom]? → [proximate cause]
2. Why [proximate cause]? → [deeper cause]
3. Why [deeper cause]? → [root cause]
```

**Category**: [Missing abstraction / Wrong assumption / Missing constraint / etc.]

### Execution Trace

The path from root cause to visible symptom:
```
[root cause] → [intermediate effect] → [intermediate effect] → [visible symptom]
```

### Blast Radius

| Affected area | Same root cause? | Severity |
|---------------|-----------------|----------|
| [File/module] | Yes / Similar pattern | High / Medium / Low |

(Or "Isolated — no other code affected")

### Fix

**Primary fix:**
```
[exact code change with file:line reference]
```

**Guard against recurrence:**
- [ ] Test: [specific test to add]
- [ ] Constraint: [type/schema/validation to add]
- [ ] Documentation: [learnings entry to create]

**Fix risk**: Safe / Low / Medium (explain what could break)
**Fix scope**: One-liner / Small / Medium
**Follows conventions**: [Verified against CLAUDE.md]

### Verification Plan
1. [How to verify the fix resolves the symptom]
2. [How to verify no regression]
3. [How to verify blast radius is addressed]

### Next Steps
- If the fix is 1-2 files with a clear change: apply directly, then `/rem-verify`
- If the fix requires coordinated changes across 3+ files, migrations, or new patterns: `/rem-plan` — pass the root cause category as context so the plan's Riskiest Assumption section captures the architectural finding
- Run `/rem-learn` to capture this bug and its root cause for future reference
- Run `/rem-review-code` on the fix to verify it doesn't introduce new issues
- If the root cause is a missing pattern: consider adding it to CLAUDE.md via `/revise-claude-md`; if it matches the gaps in `feedback_plan_vs_reality_gaps.md`, append a confirmation date there

---

## Rules

1. **Never guess.** Every claim about the root cause must be supported by evidence from the code, logs, or data. "I think it might be X" is not a diagnosis — read the code and verify.

2. **Never stop at the proximate cause.** "The query failed" is not a root cause. WHY did the query fail? WHY was the data wrong? Keep going until you reach a design or architectural issue.

3. **Check learnings first.** Don't spend 30 minutes investigating a bug that's already documented with a fix. Check existing knowledge before starting fresh investigation.

4. **Eliminate hypotheses systematically.** Don't jump to the first plausible explanation. Generate multiple hypotheses, rank them, and investigate in order. Evidence eliminates hypotheses — not intuition.

5. **Find ALL instances.** If the bug is a pattern (not a one-off typo), search the codebase for every instance of the same pattern. One fix should address all of them.

6. **Fix the root cause, not the symptom.** Adding a null check where the value should never be null is symptom-patching. Finding out WHY the value is null and fixing THAT is root-cause fixing.

7. **Respect existing conventions.** The fix must follow project conventions. Check CLAUDE.md and learnings.md before proposing a fix. Don't introduce a new pattern to fix one bug.

8. **Always suggest a recurrence guard.** The best fix prevents the entire CLASS of bug, not just this one instance. A test, a type constraint, a validation rule, or a documentation entry.

9. **3-attempt reset rule.** If you've proposed 3 fixes and none resolved the bug, STOP. Do not propose fix #4. Return to Step 1 — the architecture itself is likely the root cause, not the symptom you're chasing. Flag this explicitly: "THREE ATTEMPTS FAILED — reconsidering whether this is an architectural problem, not a localized bug." This prevents the common trap of increasingly desperate patches that each fail for a different reason.

10. **Use pattern analysis before hypotheses.** Step 1c exists for a reason — comparing working code to broken code is faster and more reliable than reasoning from first principles. If a working equivalent exists, read it first.