Agent Fault Tolerance

Agent Fault Tolerance

Implementing fault tolerance in agent systems — from redundancy and health checks through graceful degradation, recovery, and self-healing.

When to Use

• Production agent systems that must be reliable
• Multi-agent swarms where individual agents may fail
• Long-running agent workflows that must survive failures
• Mission-critical agent deployments

Fault Tolerance Patterns

FAULT_TOLERANCE_PATTERNS = {
    'redundancy': 'Multiple agents with same capability; if one fails, others take over',
    'checkpointing': 'Regularly save agent state so it can resume after failure',
    'heartbeat_monitoring': 'Health check pings; failed agents are detected and replaced',
    'circuit_breaker': 'Stop sending to failing agent, retry after cooldown',
    'degradation': 'System continues with reduced functionality when agents fail',
    'retry_with_backoff': 'Retry failed operations with exponential backoff',
}

class AgentHealthMonitor:
    """Monitor agent health and trigger recovery."""
    def __init__(self, timeout: int = 30):
        self.agents = {}  # agent_id -> last_heartbeat
        self.timeout = timeout
    
    def record_heartbeat(self, agent_id: str):
        self.agents[agent_id] = time.time()
    
    def check_health(self, agent_id: str) -> bool:
        last = self.agents.get(agent_id)
        if last is None: return False
        return (time.time() - last) < self.timeout
    
    def get_failed_agents(self) -> List[str]:
        return [a for a in self.agents if not self.check_health(a)]

Common Pitfalls

1. Insufficient redundancy — having only 2 agents for N tasks guarantees overload on failure
2. No isolation — cascade failures when one agent's failure triggers others
3. Recovery without testing — recovery paths often fail in production; test them
4. State loss — agent state not persisted; checkpoint critical state
5. Silent failures — agent fails without alerting; monitor liveness actively

Verification Checklist

• Health monitoring with heartbeat/keepalive
• Redundancy factor defined (N+1 or 2N)
• State checkpointing implemented
• Failure detection time < recovery time
• Graceful degradation path defined
• Recovery procedures tested (chaos engineering)