Resilience Patterns

Resilience Patterns

Bulkhead isolation, timeout policies, fallback strategies, and hedging patterns for NodeJS-Starter-V1.

Metadata

Description

Codifies resilience patterns for NodeJS-Starter-V1: bulkhead isolation to contain failures, timeout policies for external calls, fallback strategies when dependencies are unavailable, hedged requests for latency-sensitive paths, cascading failure prevention, and degraded mode operation for the AI provider layer.

When to Apply

Positive Triggers

• Isolating failures in AI provider calls from affecting other services
• Adding timeout policies to external API calls (Ollama, Anthropic, Stripe)
• Implementing fallback behaviour when a dependency is unavailable
• Preventing cascading failures across the multi-agent backend
• Building degraded mode for when AI providers are down
• Adding hedged requests for latency-critical operations

Negative Triggers

• Retry logic with exponential backoff (use retry-strategy skill)
• Rate limiting inbound requests (use rate-limiter skill)
• Process shutdown and connection draining (use graceful-shutdown skill)
• Error classification and codes (use error-taxonomy skill)

Core Principles

The Three Laws of Resilience

1. Isolate Blast Radius: A failing dependency must not consume all resources. Use bulkheads (semaphores, thread pools) to limit concurrent calls per dependency.
2. Fail Fast, Recover Slow: Timeouts must be shorter than the caller's patience. Circuit breakers open immediately on threshold, close gradually with half-open probes.
3. Always Have a Fallback: Every external call must have a degraded-but-functional alternative. Cached response, default value, or graceful error — never an unhandled crash.

Pattern 1: Bulkhead Isolation (Python)

Semaphore-Based Resource Limiting

import asyncio
from contextlib import asynccontextmanager
from collections.abc import AsyncGenerator


class Bulkhead:
    """Limits concurrent access to a resource to prevent cascade failure."""

    def __init__(self, name: str, max_concurrent: int, max_queue: int = 0) -> None:
        self.name = name
        self.semaphore = asyncio.Semaphore(max_concurrent)
        self.max_concurrent = max_concurrent
        self.max_queue = max_queue
        self._queued = 0

    @asynccontextmanager
    async def acquire(self) -> AsyncGenerator[None, None]:
        if self._queued >= self.max_queue:
            raise BulkheadFullError(
                f"Bulkhead '{self.name}' rejected: {self._queued} queued"
            )
        self._queued += 1
        try:
            await self.semaphore.acquire()
            self._queued -= 1
            yield
        finally:
            self.semaphore.release()


class BulkheadFullError(Exception):
    pass


# Configure per-dependency bulkheads
BULKHEADS = {
    "ollama": Bulkhead("ollama", max_concurrent=5, max_queue=10),
    "anthropic": Bulkhead("anthropic", max_concurrent=10, max_queue=20),
    "stripe": Bulkhead("stripe", max_concurrent=3, max_queue=5),
    "database": Bulkhead("database", max_concurrent=20, max_queue=50),
}

Usage:

async with BULKHEADS["ollama"].acquire():
    result = await ollama_provider.complete(prompt)

Project Reference: apps/backend/src/models/ollama_provider.py and anthropic.py — both call external APIs without concurrency limits. If Ollama hangs, all request threads block on it. Wrapping in bulkheads prevents one slow provider from exhausting the connection pool.

Pattern 2: Timeout Policy

Configurable Timeout Wrapper

import asyncio


class TimeoutPolicy:
    """Enforce maximum execution time for async operations."""

    def __init__(self, seconds: float, name: str = "") -> None:
        self.seconds = seconds
        self.name = name

    async def execute(self, coro):
        """Execute coroutine with timeout."""
        try:
            return await asyncio.wait_for(coro, timeout=self.seconds)
        except asyncio.TimeoutError:
            raise TimeoutPolicyError(
                f"Operation '{self.name}' timed out after {self.seconds}s"
            )


class TimeoutPolicyError(Exception):
    pass


# Per-dependency timeout configuration
TIMEOUTS = {
    "ollama_complete": TimeoutPolicy(30.0, "ollama_complete"),
    "ollama_embed": TimeoutPolicy(10.0, "ollama_embed"),
    "anthropic_complete": TimeoutPolicy(60.0, "anthropic_complete"),
    "stripe_api": TimeoutPolicy(15.0, "stripe_api"),
    "database_query": TimeoutPolicy(5.0, "database_query"),
}

Rule: Timeout values should decrease as you move closer to the user. Edge (3s) < API gateway (10s) < Backend service (30s) < AI provider (60s). This prevents upstream callers from waiting on already-expired downstream requests.

Pattern 3: Fallback Strategy

Graceful Degradation for AI Providers

from typing import Callable, TypeVar

T = TypeVar("T")


class FallbackChain:
    """Execute operations with fallback alternatives."""

    def __init__(self, name: str) -> None:
        self.name = name
        self.strategies: list[tuple[str, Callable]] = []

    def add(self, name: str, fn: Callable) -> "FallbackChain":
        self.strategies.append((name, fn))
        return self

    async def execute(self, *args, **kwargs) -> T:
        last_error = None
        for strategy_name, fn in self.strategies:
            try:
                return await fn(*args, **kwargs)
            except Exception as e:
                last_error = e
                logger.warning(
                    "fallback_triggered",
                    chain=self.name,
                    failed=strategy_name,
                    error=str(e),
                )
        raise FallbackExhaustedError(
            f"All strategies failed for '{self.name}'", last_error
        )


class FallbackExhaustedError(Exception):
    def __init__(self, message: str, cause: Exception | None = None) -> None:
        super().__init__(message)
        self.__cause__ = cause

Usage for AI provider fallback:

ai_fallback = (
    FallbackChain("ai_complete")
    .add("anthropic", lambda p: anthropic.complete(p))
    .add("ollama", lambda p: ollama.complete(p))
    .add("cached", lambda p: cache.get(f"ai:{hash(p)}"))
    .add("default", lambda p: "Service temporarily unavailable")
)

result = await ai_fallback.execute(prompt)

Project Reference: apps/backend/src/models/selector.py — the existing provider selector picks one provider. The fallback chain tries primary → secondary → cached → default, making the AI layer resilient to individual provider outages.

Pattern 4: Hedged Requests (TypeScript)

Parallel Requests with First-Response Wins

async function hedgedFetch<T>(
  fetchers: Array<() => Promise<T>>,
  delayMs = 200,
): Promise<T> {
  return new Promise((resolve, reject) => {
    let settled = false;
    const errors: Error[] = [];

    fetchers.forEach((fn, index) => {
      setTimeout(async () => {
        if (settled) return;
        try {
          const result = await fn();
          if (!settled) {
            settled = true;
            resolve(result);
          }
        } catch (err) {
          errors.push(err as Error);
          if (errors.length === fetchers.length && !settled) {
            settled = true;
            reject(new Error("All hedged requests failed"));
          }
        }
      }, index * delayMs);
    });
  });
}

Use case: When the backend supports multiple AI providers, fire the request to the primary after 0ms and the secondary after 200ms. Whichever responds first wins. Cancel the slower one if the first returns.

Pattern 5: Combined Resilience Policy

Composing Bulkhead + Timeout + Fallback

async def resilient_ai_call(prompt: str) -> str:
    """AI completion with full resilience stack."""
    try:
        async with BULKHEADS["anthropic"].acquire():
            return await TIMEOUTS["anthropic_complete"].execute(
                anthropic.complete(prompt)
            )
    except (BulkheadFullError, TimeoutPolicyError):
        # Fallback to local provider
        try:
            async with BULKHEADS["ollama"].acquire():
                return await TIMEOUTS["ollama_complete"].execute(
                    ollama.complete(prompt)
                )
        except (BulkheadFullError, TimeoutPolicyError):
            return "AI service temporarily unavailable. Please try again."

Execution order: Bulkhead (reject if full) → Timeout (cancel if slow) → Fallback (try next provider). Each layer adds protection without duplicating logic.

Complements: retry-strategy skill — wrap the entire resilient_ai_call with retry_async() for transient network errors. health-check skill — expose bulkhead utilisation and timeout rates in the deep health endpoint.

Pattern 6: Degraded Mode Configuration

Feature Availability Matrix

from enum import Enum


class ServiceStatus(str, Enum):
    HEALTHY = "healthy"
    DEGRADED = "degraded"
    UNAVAILABLE = "unavailable"


DEGRADED_MODE_CONFIG = {
    "ai_complete": {
        ServiceStatus.HEALTHY: "full_response",
        ServiceStatus.DEGRADED: "cached_or_short_response",
        ServiceStatus.UNAVAILABLE: "static_fallback_message",
    },
    "search": {
        ServiceStatus.HEALTHY: "hybrid_fts_vector",
        ServiceStatus.DEGRADED: "fts_only",
        ServiceStatus.UNAVAILABLE: "title_match_only",
    },
    "embeddings": {
        ServiceStatus.HEALTHY: "generate_new",
        ServiceStatus.DEGRADED: "use_cached",
        ServiceStatus.UNAVAILABLE: "skip_embedding",
    },
}

Expose the current mode via the health endpoint so the frontend can adjust its UI accordingly (e.g., show "Limited AI features" banner when degraded).

Anti-Patterns

Checklist

Before merging resilience-patterns changes:

• Bulkheads configured per external dependency with max_concurrent limits
• Timeout policies set per operation type (AI > API > DB)
• Fallback chain for AI provider layer (primary → secondary → cached → default)
• Hedged requests for latency-critical paths
• Combined policy: bulkhead → timeout → fallback composition
• Degraded mode configuration for each service capability
• Health endpoint exposes bulkhead utilisation and circuit breaker state
• BulkheadFullError returns 503 Service Unavailable, not 500

Response Format

When applying this skill, structure implementation as:

### Resilience Implementation

**Bulkheads**: [per-dependency / global], limits=[ollama:5, anthropic:10, db:20]
**Timeouts**: [per-operation], values=[ai:60s, api:15s, db:5s]
**Fallback**: [chain / single / none], strategies=[primary, secondary, cached, default]
**Hedging**: [enabled / disabled], delay=[200ms]
**Degraded Mode**: [configured / not configured]
**Integration**: [retry-strategy / health-check / both]