manastalukdar Chaos Test — Resilience Gap Analysis
Scan the codebase for failure modes that would surface under real-world stress: network partitions, dependency outages, slow consumers, and burst traffic.
Usage
/chaos-test # full scan of staged changes and key integration points
/chaos-test <path> # target a specific file or directory
/chaos-test --service <name> # focus on a named external service dependency
Behavior
Step 1 — Identify integration boundaries
# Find outbound HTTP calls, queue producers/consumers, DB calls, cache hits
grep -rn "fetch\|axios\|http\.\|requests\.\|redis\|pg\.\|mysql\|amqp\|kafka" \
--include="*.ts" --include="*.py" --include="*.go" --include="*.js" \
-l . | head -30
Step 2 — Check each boundary for resilience controls
For every integration point found, verify presence of:
| Control | What to grep for | Risk if absent |
|---|---|---|
| Timeout | timeout, AbortController, signal |
Hangs under slow upstream |
| Retry | retry, backoff, attempt |
Single transient failure = hard failure |
| Circuit breaker | opossum, cockatiel, breaker, circuitbreaker |
Cascading failure under sustained outage |
| Fallback / degraded mode | fallback, default, catch returning a stub |
No graceful degradation |
| Rate-limit handling | 429, RateLimitError, retry-after |
Thundering herd kills upstream |
| Bulkhead / concurrency cap | semaphore, pLimit, asyncio.Semaphore |
Single slow dep starves thread pool |
Step 3 — Score each boundary
For each integration point, assign a risk tier:
- Critical — no timeout AND no retry AND no fallback
- High — missing two of the three controls above
- Medium — missing one control
- Low — all controls present; note any configuration concerns (timeout too high, retry without jitter, etc.)
Step 4 — Report findings
Output a prioritized list:
CHAOS ANALYSIS — <path>
Critical (fix before next deploy)
src/payments/stripe.ts:42 POST /charges — no timeout, no retry, no fallback
src/auth/token.ts:88 Redis GET — no timeout; outage causes auth failure
High
src/search/elastic.ts:17 ES query — no circuit breaker; slow queries block response
Medium
src/email/sendgrid.ts:31 Retry present but no jitter (thundering herd risk)
Low
src/cache/redis.ts:55 All controls present; timeout at 30s is high — consider 5s
Suggested experiment targets (fault to inject → expected behavior → gap if wrong):
1. Kill Redis → auth should degrade to DB lookup → currently: hard 500
2. Throttle Stripe to 2s → checkout should timeout at 1s → currently: no timeout set
Step 5 — Suggest quick wins
For each Critical/High finding, propose the minimal fix:
stripe.ts:42 — Add AbortSignal timeout (5s) and exponential-backoff retry (3 attempts, 100ms base, jitter):
const res = await fetch(url, { signal: AbortSignal.timeout(5000) })
Edge Cases
- No external dependencies: Report "No integration boundaries detected" and exit.
- Monorepo: Scan from
$ARGUMENTSpath; skip test files unless--include-testspassed. - gRPC / GraphQL: Detect
grpc,graphql-request, andapollo-clientas integration points. - Already uses resilience library: Note which library is in use and check it is configured, not just imported.
Token Optimization
Expected range: 600–2,000 tokens (grep-driven discovery); 100–300 tokens (no integrations found, early exit)
Patterns used: Grep-before-Read (scan for integration points before reading any file), git diff scope (staged changes first), progressive disclosure (summary → details on request)
Early exit: If git diff --staged --name-only returns no files and no $ARGUMENTS path is given, report "No staged changes to analyze" and exit.
Caching: Caches file list from boundary scan in .claude/cache/chaos-test/boundaries.json (invalidated when package.json, go.mod, or requirements.txt changes).