Ductile Rca

Ductile RCA

This skill is Pillar 3: RCA of the Ductile Constitution (see ../../CONSTITUTION.md) — the thinking discipline for incidents. It is not a recipe book. The recipe is in ductile (Pillar 1) and the fix is in ductile-plugin-developer (Pillar 5). What this skill insists on is what you think before you type.

Seam: relationship to other skills

Co-invocation matrix:

Out of scope

• Writing the fix (→ ductile-plugin-developer)
• Deploying the fix (→ ductile)
• Routine inspection of a healthy system (→ ductile)

Step 1 — What is your problem?

"Programmers know the benefits of everything and the tradeoffs of nothing." The faster you reach for a tool, the longer the incident.

Before any command runs, fill this in. If you cannot fill it in, you are not yet investigating — you are flailing.

SYMPTOM (what is observably wrong):
  …

EXPECTATION (what should be happening instead):
  …

WHEN DID IT START (timestamp, last known good):
  …

WHO IS AFFECTED (which instance, which plugin, which pipeline, which user):
  …

WHAT CHANGED RECENTLY (deploy, config lock, plugin update, host event):
  …

WHAT YOU HAVE ALREADY VERIFIED (and the evidence for it):
  …

Refuse to proceed until this is named. The shape of the problem statement tells you which evidence to pull next; the wrong shape leads to noise.

Step 2 — Hammock first. Do not instrument yet.

Hickey's hammock-driven development applies precisely here. Think for ten minutes before you type ten commands. List the candidate hypotheses before collecting evidence — otherwise you confirm the first one you stumble into.

Write down 2-5 hypotheses ranked by prior probability:

H1: <most likely cause given the symptom shape>
H2: <next most likely>
H3: <wildcard — what would I not want to be true?>

For each hypothesis, write the discriminating observation: the single piece of evidence that, once collected, would distinguish this hypothesis from the others. That is what you collect first.

Step 3 — Evidence in non-faulty-entity order

"Errors must be detected by some non-faulty entity." — Armstrong

The plugin is the faulty entity. Its self-report is the last thing you consult, not the first. Walk this ladder in order; do not skip rungs:

The rule: never quote plugin stderr as root cause before you have confirmed the gateway's view (rungs 1-5). A plugin that reports success to a job the gateway recorded as failed is more interesting than either fact alone.

Step 4 — Hypothesis testing: why-this-but-not-that

For each surviving hypothesis, force two questions:

1. Why this? — what is the observation that positively implies this hypothesis (not merely is consistent with it)?
2. Why not that? — name the sibling hypotheses you are now rejecting and the evidence that rules them out.

If you cannot answer "why not that" for the alternatives, you have a guess, not a diagnosis. Keep collecting until you can.

Step 5 — Blast radius: what-could-be-impacted-but-isn't

Hickey: values vs places. A failed plugin_fact is a value at a point in time — it does not retroactively corrupt the past. But the places that project from that fact (compatibility view, downstream pipeline state, sibling-container DB) can drift.

For every confirmed failure, ask:

• Which pipelines uses: this plugin? (ductile config show pipeline:*)
• Which webhooks subscribe to events this plugin emits?
• Which sibling systems read from a DB this plugin writes to via another writer (e.g., the garmin container writes garmin.db; the ductile plugin is only a mtime watcher)?
• Is the circuit breaker open, masking downstream symptoms?

If the blast radius is wider than the symptom, you may be looking at the effect, not the cause.

Step 6 — Validate before you fix

For each candidate fix, design one cheap test that would confirm it.

If the cheap test cannot distinguish your hypothesis from the alternatives, you do not yet have the test. Refine before acting.

Common ductile failure modes (with evidence pattern)

The recurring root cause(s): forgot to lock — config lock or plugin lock

Half of "why isn't my change live?" is a lock not run. Since the two locks decoupled, there are two H1s — pick by what changed.

After a config file edit (config.yaml, api.yaml, pipelines/*.yaml, webhooks.yaml):

ductile config check          # validate
ductile config lock           # authorize the file state
ductile system reload         # apply

After a plugin manifest/entrypoint edit — the bytes are attested, so this is the sibling H1, easy to miss post-decoupling:

ductile plugin lock <name>    # re-attest the changed bytes
ductile system reload

config lock does not re-attest plugin bytes. The tell for the plugin-lock H1: a secret-consuming plugin started failing or lost its secrets right after an edit, and the operator ran config lock but not plugin lock — the spawn-time fingerprint check then refuses its secrets (fail closed). If the user describes a change followed by no apparent effect, one of these two is the hypothesis every time.

Handoff back to the other skills

• Fix is in plugin code → ductile-plugin-developer
• Fix is in config / lock / deploy → ductile
• Fix is "operator forgot a step" — close the incident with a runbook update in docs/OPERATOR_GUIDE.md (idiom 8: every surface is agent-drivable).

References

• Global rca skill — generic discipline this skill extends
• Global diagnose skill — when there is no running gateway
• docs/PLUGIN_DIAGNOSTICS.md — structured triage commands
• docs/reload_rca.md — worked RCA example, canonical pattern
• docs/HEALTH_CHECK.md — invariants checked by selfcheck
• docs/SQL_TIGHTENING_LOG.md — schema-change audit trail (when "what changed?" is "the schema")
• docs/SECRETS.md — the vault + attestation model (for secret-delivery and fingerprint-mismatch incidents)