Tune load balancer health checks
A health check is a contract with the balancer about what "able to serve" means. Too shallow and it routes traffic to a broken node; too strict and it churns nodes out of the pool under load.
Procedure
- Pick a dedicated endpoint that touches real dependencies but is cheap:
The check must fail when the database or cache the app needs is unreachable, and pass otherwise.curl -sf -o /dev/null -w '%{http_code} %{time_total}\n' http://10.0.1.5:8080/healthz - Measure the endpoint's own latency, then set the timeout above its p99:
Setfor i in $(seq 1 50); do curl -s -o /dev/null -w '%{time_total}\n' http://localhost:8080/healthz; done \ | sort -n | tail -1 # slowest of 50 ≈ p98timeout = 2 × slowest. A timeout below p99 causes flapping exactly when traffic is high. - Choose interval and unhealthy threshold from the drain budget:
interval=10s,unhealthy_threshold=3→ 30 s to detect a dead node. Largerintervaldetects slower but generates less check traffic and jitter.
- Set a separate healthy threshold (usually 2) so one flaky success does not re-add a bad node.
- Return
503from the endpoint when a dependency is down, not a200with a JSON body — the LB reads the status code, not your payload. - Exclude the health endpoint from auth and rate limits so the checker never gets a 401/429.
- After changing thresholds, re-check the balancer's reported state:
aws elbv2 describe-target-health --target-group-arn "$TG" \ --query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State,TargetHealth.Reason]'
Pitfalls
- A health check that only dials the TCP port ("connection refused = down") passes while the app returns 500 on every real request.
- A check that pings
/(full page) hits the DB on every probe and times out under load. timeoutshorter than the endpoint's p99 flaps nodes out during traffic spikes, shrinking capacity exactly when you need it.- A
200 OKreturned by a catch-all error handler masks a dead dependency. - Thresholds set per target group but a second target group (the one for HTTP/2 or a different port) left at defaults.
- Health endpoint behind the same auth as the API returns 401 forever, marking everything down.
Verification
curl -sf -o /dev/null -w '%{http_code} %{time_total}\n' http://10.0.1.5:8080/healthz
aws elbv2 describe-target-health --target-group-arn "$TG" \
--query 'TargetHealthDescriptions[].TargetHealth.State'
Pass means the endpoint returns 200 in under the configured timeout and every target reads healthy.
Report: "Timeout raised 1s→5s (p99 2.1s), interval 10s, unhealthy 3; flapping stopped."