# Distributed Systems Debugging

> Debug failures involving retries, timeouts, partial success, replication, failover, consistency, ordering, recovery, and distributed state. Use when behavior spans services, nodes, regions, or asynchronous boundaries.

- Skill: `iamhrithikraj/distributed-systems-debugging` (Agent Skill)
- Install (CLI): `npx skillmds@latest add iamhrithikraj/distributed-systems-debugging`
- Raw SKILL.md: https://api.skillmd.com/api/skills/iamhrithikraj/distributed-systems-debugging/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: iamHrithikRaj (https://skillmd.com/u/iamhrithikraj)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/iamhrithikraj/distributed-systems-debugging

---


# Distributed systems debugging

## Define the invariant

State the property that should always hold, such as:

- An acknowledged write remains durable.
- A request is applied at most once.
- Failover does not expose stale ownership.
- Recovery converges without operator intervention.

Describe the observed counterexample precisely.

## Map the flow

Identify:

- State owners and durable stores.
- Control-plane and data-plane paths.
- Synchronous and asynchronous boundaries.
- Retry, timeout, cancellation, and failover behavior.
- Replication, quorum, ordering, and consistency assumptions.

## Build an evidence table

For each signal capture:

- Timestamp and clock source.
- Component and correlation identifier.
- Observation.
- Reliability and possible ambiguity.
- Hypotheses supported or contradicted.

## Test hypotheses

- Rank hypotheses by explanatory power, not familiarity.
- Design tests that distinguish between leading hypotheses.
- Check duplicate delivery, late completion, stale reads, reordered events,
  partial persistence, and split ownership.
- Consider whether local success can hide global failure.

## Recommend changes

- Restore the invariant at the narrowest responsible boundary.
- Make retries and recovery explicit and observable.
- Avoid adding coordination unless the invariant requires it.
- Include telemetry and regression coverage needed to detect recurrence.