Diagnose Before Retry

Diagnose Before Retry

A blind retry of a failed op reproduces the failure and wastes a cycle. A diagnosed retry tests a hypothesis and either confirms the fix or refines it.

The move

1. Don't retry immediately. The urge to re-run is strong; resist it. A failure is information — read it first.
2. Read the relevant log/evidence. The op's own output, the service's server log, the exit code, the state it left behind. Find the failure line and what preceded it.
3. Name how far the op got. "Created the migrations table, then lost connectivity mid-run." This matters for idempotency — a partially-applied migration may need rollback, not a naive retry.
4. Distinguish op-caused failure from self-caused artifact. Your own TaskStop / kill / cancellation shows up in logs as a disconnect or EOF. Don't misdiagnose your own stop as an infrastructure failure:

   "Postgres itself is healthy — the disconnect line is my TaskStop."

5. Form a hypothesis with supporting context + appropriate uncertainty.

   "The failure may have been transient load (three parallel jobs at the time)." Name the cause, the supporting context, and use uncertainty language ("may have been", "likely") — don't assert certainty you can't back.

6. Retry in a way that tests the hypothesis. If the hypothesis is "transient load from parallel jobs", retry in the foreground with no parallel load. The retry both attempts the op and tests the hypothesis.

The retry-with-observability idiom

timeout <N> <command> 2>&1 | tr '\r' '\n' | grep -vE "^\s*$" | tail -15

• Foreground, not backgrounded — direct view for a retry you care about.
• timeout <N> — bound it so a hung retry can't hang forever (the original failure mode may recur).
• tr '\r' '\n' — progress bars use carriage returns; converting to newlines stops them smearing into one unreadable line.
• grep -vE "^\s*$" — strip blank lines.
• tail -15 — bound the output; the last 15 lines show the result.

When a blind retry IS okay

• The failure was a known-transient external blip (a flaky network, a registry 5xx) and the op is idempotent and cheap.
• The op has no partial state (a read-only command, a fresh build to a clean dir).
• You've already diagnosed the same failure mode before in this session and confirmed the retry strategy.

If any of those is false, diagnose first.

Fix the structural cause, not the symptom

When a retry fixes the symptom, ask why the failure happened and fix the structure so it can't recur:

"Corpora now travel with the repo — the build can't miss them."

A prior build missed the corpora; instead of patching the build to fetch them, move the corpora into the repo so the build structurally can't miss them. The fix makes the failure mode impossible, not just unlikely. This is the highest form of diagnosis: change the structure so the bug can't recur, rather than adding a check for the symptom. A symptom-check fix leaves the root cause; a structural fix removes it.

Anti-patterns

• Blind retry. "It failed, let me run it again." Reproduces the failure; teaches nothing.
• Misdiagnosing self-caused artifacts. Your own TaskStop shows as a disconnect; "Postgres is down" → you "fix" healthy infra.
• Certainty you can't back. "It was definitely transient load." You don't know; say "may have been".
• Unbounded retry. No timeout → a hung retry hangs forever.
• Backgrounding the retry. You care about this one; foreground it for direct view.
• Ignoring how far it got. A partially-applied migration retried naively can double-apply or conflict.

Pair with

• silent-op-recovery — the retry-with-observability idiom extends that skill's capture-exit-and-sample.
• background-failure-triage — the failure that triggered the diagnosis was likely a backgrounded op.
• correct-assumptions — "the documented blocker was wrong" is one possible diagnosis.
• cost-transparency — a second retry of a 7m op should be diagnosed, not blindly re-run.