# Resilience Patterns > Resilience Patterns Skill - Skill: `yonatangross/resilience-patterns` (Agent Skill, multi-file: 15 files) - Install (CLI): `npx skillmds@latest add yonatangross/resilience-patterns` - Raw SKILL.md: https://api.skillmd.com/api/skills/yonatangross/resilience-patterns/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: yonatangross (https://skillmd.com/u/yonatangross) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/yonatangross/resilience-patterns --- # Resilience Patterns Skill Production-grade resilience patterns for distributed systems and LLM-based workflows. Covers circuit breakers, bulkheads, retry strategies, and LLM-specific resilience techniques. ## When to Use This Skill - Building fault-tolerant multi-agent systems - Implementing LLM API integrations with proper error handling - Designing distributed workflows that need graceful degradation - Adding observability to failure scenarios - Protecting systems from cascade failures ## Core Patterns ### 1. Circuit Breaker Pattern (reference: circuit-breaker.md) Prevents cascade failures by "tripping" when a service exceeds failure thresholds. ``` ┌─────────────────────────────────────────────────────────────────┐ │ Circuit Breaker States │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────┐ failures >= threshold ┌──────────┐ │ │ │ CLOSED │ ─────────────────────────▶ │ OPEN │ │ │ │ (normal) │ │ (reject) │ │ │ └────┬─────┘ └────┬─────┘ │ │ │ │ │ │ │ success timeout │ │ │ │ expires │ │ │ │ ┌────────────┐ │ │ │ │ │ HALF_OPEN │◀────────────────┘ │ │ └─────────│ (probe) │ │ │ └────────────┘ │ │ │ │ CLOSED: Allow requests, count failures │ │ OPEN: Reject immediately, return fallback │ │ HALF_OPEN: Allow probe request to test recovery │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` **Key Configuration:** - `failure_threshold`: Failures before opening (default: 5) - `recovery_timeout`: Seconds before attempting recovery (default: 30) - `half_open_requests`: Probes to allow in half-open (default: 1) ### 2. Bulkhead Pattern (reference: bulkhead-pattern.md) Isolates failures by partitioning resources into independent pools. ``` ┌─────────────────────────────────────────────────────────────────┐ │ Bulkhead Isolation │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │ TIER 1: Critical │ │ TIER 2: Standard │ │ │ │ (5 workers) │ │ (3 workers) │ │ │ │ ┌─┐ ┌─┐ ┌─┐ │ │ ┌─┐ ┌─┐ ┌─┐ │ │ │ │ │█│ │█│ │░│ │ │ │█│ │░│ │░│ │ │ │ │ └─┘ └─┘ └─┘ │ │ └─┘ └─┘ └─┘ │ │ │ │ ┌─┐ ┌─┐ │ │ │ │ │ │ │░│ │░│ │ │ Queue: 2 │ │ │ │ └─┘ └─┘ │ │ │ │ │ │ Queue: 0 │ └──────────────────┘ │ │ └──────────────────┘ │ │ │ │ ┌──────────────────┐ │ │ │ TIER 3: Optional │ █ = Active request │ │ │ (2 workers) │ ░ = Available slot │ │ │ ┌─┐ ┌─┐ │ │ │ │ │█│ │█│ FULL! │ Tier 1: synthesis, quality_gate │ │ │ └─┘ └─┘ │ Tier 2: analysis agents │ │ │ Queue: 5 │ Tier 3: enrichment, optional features │ │ └──────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` **Tier Configuration (real-world applications):** | Tier | Workers | Queue | Timeout | Use Case | |------|---------|-------|---------|----------| | 1 (Critical) | 5 | 10 | 300s | Synthesis, quality gate | | 2 (Standard) | 3 | 5 | 120s | Content analysis agents | | 3 (Optional) | 2 | 3 | 60s | Enrichment, caching | ### 3. Retry Strategies (reference: retry-strategies.md) Intelligent retry logic with exponential backoff and jitter. ``` ┌─────────────────────────────────────────────────────────────────┐ │ Exponential Backoff + Jitter │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ Attempt 1: ──▶ X (fail) │ │ wait: 1s ± 0.5s │ │ │ │ Attempt 2: ──▶ X (fail) │ │ wait: 2s ± 1s │ │ │ │ Attempt 3: ──▶ X (fail) │ │ wait: 4s ± 2s │ │ │ │ Attempt 4: ──▶ ✓ (success) │ │ │ │ Formula: delay = min(base * 2^attempt, max_delay) * jitter │ │ Jitter: random(0.5, 1.5) to prevent thundering herd │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` **Error Classification for Retries:** ```python RETRYABLE_ERRORS = { # HTTP/Network 408, 429, 500, 502, 503, 504, # HTTP status codes ConnectionError, TimeoutError, # Network errors # LLM-specific "rate_limit_exceeded", "model_overloaded", "context_length_exceeded", # Retry with truncation } NON_RETRYABLE_ERRORS = { 400, 401, 403, 404, # Client errors "invalid_api_key", "content_policy_violation", "invalid_request_error", } ``` ### 4. LLM-Specific Resilience (reference: llm-resilience.md) Patterns specific to LLM API integrations. ``` ┌─────────────────────────────────────────────────────────────────┐ │ LLM Fallback Chain │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ Request ──▶ [Primary Model] ──success──▶ Response │ │ │ │ │ fail │ │ ▼ │ │ [Fallback Model] ──success──▶ Response │ │ │ │ │ fail │ │ ▼ │ │ [Cached Response] ──hit──▶ Response │ │ │ │ │ miss │ │ ▼ │ │ [Default Response] ──▶ Graceful Degradation │ │ │ │ Example Chain: │ │ 1. claude-sonnet-4-20250514 (primary) │ │ 2. gpt-4o-mini (fallback) │ │ 3. Semantic cache lookup │ │ 4. "Analysis unavailable" + partial results │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` **Token Budget Management:** ``` ┌─────────────────────────────────────────────────────────────────┐ │ Token Budget Guard │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ Input: 8,000 tokens │ │ ┌─────────────────────────────────────────────────┐ │ │ │████████████████████████████░░░░░░░░░░░░░░░░░░░░│ │ │ └─────────────────────────────────────────────────┘ │ │ ▲ │ │ │ │ │ Context Limit (16K) │ │ │ │ Strategy when approaching limit: │ │ 1. Summarize earlier context (compress 4:1) │ │ 2. Drop low-priority content (optional fields) │ │ 3. Split into multiple requests │ │ 4. Fail fast with "content too large" error │ │ │ └─────────────────────────────────────────────────────────────────┘ ``` ## Quick Reference | Pattern | When to Use | Key Benefit | |---------|-------------|-------------| | Circuit Breaker | External service calls | Prevent cascade failures | | Bulkhead | Multi-tenant/multi-agent | Isolate failures | | Retry + Backoff | Transient failures | Automatic recovery | | Fallback Chain | Critical operations | Graceful degradation | | Token Budget | LLM calls | Cost control, prevent failures | ## real-world applications Integration Points 1. **Workflow Agents**: Each agent wrapped with circuit breaker + bulkhead tier 2. **LLM Calls**: All model invocations use fallback chain + retry logic 3. **External APIs**: Circuit breaker on YouTube, arXiv, GitHub APIs 4. **Database Ops**: Bulkhead isolation for read vs write operations ## Files in This Skill ### References (Conceptual Guides) - `references/circuit-breaker.md` - Deep dive on circuit breaker pattern - `references/bulkhead-pattern.md` - Bulkhead isolation strategies - `references/retry-strategies.md` - Retry algorithms and error classification - `references/llm-resilience.md` - LLM-specific patterns - `references/error-classification.md` - How to categorize errors ### Templates (Code Patterns) - `templates/circuit-breaker.py` - Ready-to-use circuit breaker class - `templates/bulkhead.py` - Semaphore-based bulkhead implementation - `templates/retry-handler.py` - Configurable retry decorator - `templates/llm-fallback-chain.py` - Multi-model fallback pattern - `templates/token-budget.py` - Token budget guard implementation ### Examples - `examples/project-workflow-resilience.md` - Full real-world applications integration example ### Checklists - `checklists/pre-deployment-resilience.md` - Production readiness checklist - `checklists/circuit-breaker-setup.md` - Circuit breaker configuration guide ## 2025 Best Practices 1. **Adaptive Thresholds**: Use sliding windows, not fixed counters 2. **Observability First**: Every circuit trip = alert + metric + trace 3. **Graceful Degradation**: Always have a fallback, even if partial 4. **Health Endpoints**: Separate health check from circuit state 5. **Chaos Testing**: Regularly test failure scenarios in staging