Circuit Breaker Tuner

Circuit Breaker Tuner

When to Use / When Not to Use

Use when:

• A slow or failing downstream service is exhausting threads or connection pools
• You need to tune failure thresholds, recovery timing, or HALF_OPEN probe behavior
• An application has no resilience layer and a downstream failure takes down the whole system
• You need fallback strategies for graceful degradation

Do not use when:

• The problem is slow SQL queries or missing database indexes
• You need chaos experiment design (use chaos-engineer)

Process

1. Identify the dependency — Which downstream service is failing or slow? What is its typical latency and recovery time?
2. Choose state machine parameters — Set failureRateThreshold, waitDurationInOpenState, minimumNumberOfCalls
3. Configure the sliding window — COUNT_BASED (by request count) vs TIME_BASED (by time window)
4. Define a fallback — What should callers receive when the circuit is OPEN?
5. Add a bulkhead — Limit concurrent calls to prevent pool exhaustion before the circuit opens
6. Wire monitoring — Alert on circuit state transitions (CLOSED → OPEN)

For full YAML templates, see references/resilience4j-config.md. For fallback code examples, see references/fallback-patterns.md. For metrics, see references/metrics-alerting.md.

Output Template

For each circuit breaker configuration, provide:

1. Resilience4j YAML or code config with commented rationale
2. Fallback method implementation
3. Bulkhead config (concurrent call limit)
4. Monitoring alert rule for state change events
5. Test scenario verifying OPEN → HALF_OPEN → CLOSED recovery

What Claude Does / What You Do

The State Machine

CLOSED → (failures exceed threshold) → OPEN → (wait duration) → HALF_OPEN
HALF_OPEN → (probe succeeds) → CLOSED
HALF_OPEN → (probe fails) → OPEN

• CLOSED: Normal operation. Failures are counted.
• OPEN: Failing fast. All calls return error/fallback immediately. Downstream gets time to recover.
• HALF_OPEN: Probe state. A limited number of calls test recovery. Success → CLOSED; failure → OPEN.

Setting Each Threshold

Starting point: 50%. Tune down if circuit fails to open when it should; tune up if it opens during brief spikes.

waitDurationInOpenState: Set slightly longer than the downstream service's typical restart or recovery time. 5–15s for fast-recovering services; 60–120s for services needing operator intervention.

minimumNumberOfCalls: Start at 10. Setting to 1–2 causes a single slow call to open the circuit.

Fallback Strategies

Common Mistakes

• minimumNumberOfCalls too low — single slow call opens the circuit
• waitDurationInOpenState too short — circuit cycles open/probe/open without recovery
• Client errors (4xx) not in ignoreExceptions — inflate failure rate with caller mistakes
• Same timeout for all services — payment and recommendation have different profiles
• No fallback defined — OPEN state throws uncaught exception instead of degrading gracefully
• HTTP timeout < slowCallDurationThreshold — calls time out before they're counted as slow

Quick Checklist

• minimumNumberOfCalls ≥ 10
• failureRateThreshold calibrated for this specific dependency
• slowCallDurationThreshold consistent with HTTP client read timeout
• waitDurationInOpenState longer than downstream recovery time
• 4xx errors in ignoreExceptions
• Fallback method defined with meaningful degraded behavior
• Bulkhead configured alongside circuit breaker
• Circuit state transitions trigger monitoring alerts

Related Skills

• connection-pool-tuner — pool exhaustion often co-occurs with missing circuit breakers
• chaos-engineer — test circuit breaker behavior with controlled failure injection
• transaction-boundary-reviewer — wide transactions compound cascading failure risk