Health Check Patterns

Health Check Patterns

Comprehensive health check patterns for cloud-native applications, implementing Kubernetes probes (liveness, readiness, startup), HTTP health endpoints, database connectivity checks, external service monitoring, and circuit breaker patterns to ensure application reliability and service availability.

TL;DR Checklist

• Implement separate liveness and readiness probes with distinct purposes
• Configure appropriate probe timeouts and thresholds for slow-starting applications
• Add database connection health checks with connection pool metrics
• Implement HTTP health endpoint with structured JSON response
• Monitor external service dependencies with timeout and retry logic
• Add circuit breaker patterns for failing external dependencies
• Test health checks in isolation before deployment
• Set up alerting based on health check failures with appropriate thresholds

When to Use

Use health check patterns when:

• Deploying applications to Kubernetes or other container orchestration platforms
• Building microservices that need to expose health status for load balancers
• Implementing distributed systems with multiple external dependencies
• Creating resilient applications that must fail gracefully when dependencies fail
• Setting up CI/CD pipelines with automated health verification
• Designing service mesh configurations that depend on health status

When NOT to Use

Avoid these patterns for:

• Simple standalone scripts — Use basic exit codes instead of complex health endpoints
• Single-process applications without dependencies — Health checks may add unnecessary overhead
• Real-time systems with strict timing requirements — Health check overhead may impact performance
• Local development without orchestration — Use simpler logging and error handling

Core Workflow

1. Design Health Strategy — Define what "healthy" means for your application (database, cache, external services). Checkpoint: Document all dependencies and their health impact on service availability.

2. Implement Kubernetes Probes — Create separate liveness, readiness, and startup probes with appropriate configurations. Checkpoint: Verify probe endpoints respond correctly with correct HTTP status codes.

3. Build HTTP Health Endpoint — Create a standardized health endpoint that aggregates all dependency checks. Checkpoint: Test endpoint returns 200 OK when healthy and 503 Service Unavailable when degraded.

4. Add Database Health Checks — Implement connection pool monitoring with configurable timeout thresholds. Checkpoint: Verify health check fails gracefully when database is unreachable.

5. Monitor External Services — Create health checks for all external service dependencies with circuit breaker patterns. Checkpoint: Confirm circuit breaker opens when external service fails repeatedly.

6. Set Up Testing and Validation — Implement automated health check testing in CI/CD pipeline. Checkpoint: Run health check tests in staging environment matching production configuration.

Implementation Patterns

Pattern 1: Kubernetes Liveness Probes

Liveness probes determine if a container should be restarted. They should check internal application state, not external dependencies.

# ❌ BAD — liveness probe checking external dependency (should not depend on database)
livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

# ✅ GOOD — simple liveness probe checking application is still running
livenessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - "python -c 'import socket; s=socket.socket(); s.connect((\"localhost\", 8080)); s.close()'"
  initialDelaySeconds: 15
  periodSeconds: 20
  timeoutSeconds: 5
  failureThreshold: 3

Best Practice: Liveness probes should only check if the application process is responding, not if it's fully functional. Use external dependencies only for readiness probes.

Pattern 2: Kubernetes Readiness Probes

Readiness probes determine if a pod should receive traffic. They should check all critical dependencies.

# ✅ GOOD — readiness probe checking all dependencies
readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 3
  successThreshold: 1
  failureThreshold: 3

# ❌ BAD — readiness probe with incorrect thresholds
readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 0  # Too aggressive, starts receiving traffic before ready
  periodSeconds: 1  # Too frequent, creates overhead
  timeoutSeconds: 1  # Too short, may timeout on slow dependencies

Pattern 3: Kubernetes Startup Probes

Startup probes are for slow-starting applications that need more time to initialize.

# ✅ GOOD — startup probe for application with slow initialization
startupProbe:
  httpGet:
    path: /health/started
    port: 8080
  failureThreshold: 30
  periodSeconds: 10
  timeoutSeconds: 5

# ❌ BAD — startup probe missing, relying only on initialDelaySeconds
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 300  # Long delay but no startup probe

Configuration Example:

apiVersion: v1
kind: Pod
metadata:
  name: slow-start-app
spec:
  containers:
  - name: app
    image: my-app:latest
    ports:
    - containerPort: 8080
    startupProbe:
      httpGet:
        path: /health/started
        port: 8080
      failureThreshold: 60
      periodSeconds: 5
    livenessProbe:
      httpGet:
        path: /health/live
        port: 8080
      initialDelaySeconds: 60
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /health/ready
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5

Pattern 4: HTTP Health Endpoint with Structured Response

Create a standardized health endpoint that returns comprehensive status information. Use curl for testing and YAML for configuration.

# ❌ BAD — simple health check without structured response
curl -s http://localhost:8080/health | jq .
# Returns: {"status": "ok"}  — no dependency details

# ✅ GOOD — comprehensive health check with all dependency checks
# Test liveness endpoint
curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health/live

# Test readiness endpoint with full JSON response
curl -s http://localhost:8080/health/ready | jq .

# Test comprehensive health endpoint
curl -s http://localhost:8080/health | jq '
  .status,
  (.checks | keys),
  .timestamp
'

# Verify response structure contains all expected fields
curl -s http://localhost:8080/health | jq '
  if .status == "healthy" and (.checks | has("database")) and (.checks | has("redis"))
  then "✅ Health check structure valid"
  else "❌ Health check structure invalid"
  end
'

# Example structured response from healthy endpoint:
# {
#   "status": "healthy",
#   "checks": {
#     "database": { "status": "healthy", "latency_ms": 15 },
#     "redis": { "status": "healthy", "latency_ms": 5 }
#   },
#   "timestamp": "2025-01-15T10:30:00Z"
# }

Pattern 5: Database Health Checks with Connection Pool Metrics

Monitor database health with connection pool statistics for proactive issue detection. Use CLI tools for database health checks.

# ❌ BAD — basic database health check without metrics
pg_isready -h localhost -U app -d app
# Returns only: "localhost:5432 - accepting connections"

# ✅ GOOD — comprehensive database health check with metrics using psql
# Single query to check database connectivity with timing
psql -h localhost -U app -d app -c "
  SELECT 
    current_timestamp as check_time,
    pg_postmasters() AS pid,
    pg_conf_load_time() AS config_reload_time
  \\gset
"

# Check PostgreSQL connection pool metrics
psql -h localhost -U app -d app -c "
  SELECT 
    count(*) AS total_connections,
    count(*) FILTER (WHERE state = 'active') AS active_connections,
    count(*) FILTER (WHERE state = 'idle') AS idle_connections,
    count(*) FILTER (WHERE state = 'idle in transaction') AS idle_in_transaction,
    max(state_change) AS last_state_change
  FROM pg_stat_activity
  WHERE datname = 'app';
"

# Check connection pool saturation (returns 1 if >80% utilized)
psql -h localhost -U app -d app -c "
  SELECT 
    CASE 
      WHEN max_connections > 0 THEN 
        round(count(*)::numeric / max_connections * 100, 1)
      ELSE 0 
    END AS utilization_pct
  FROM pg_stat_activity;
  SELECT CASE WHEN count(*) * 100 / (SELECT setting FROM pg_settings WHERE name = 'max_connections') > 80 
    THEN '⚠️ POOL SATURATED' ELSE '✅ Pool healthy' END AS pool_status
  FROM pg_stat_activity;
"

# Check database replication lag (for read replicas)
psql -h localhost -U app -d app -c "
  SELECT 
    client_addr,
    state,
    sent_lsn,
    write_lsn,
    flush_lsn,
    replay_lsn,
    pg_wal_lsn_diff(sent_lsn, replay_lsn) AS replication_lag_bytes
  FROM pg_stat_replication;
"

# Check for long-running queries (>30s)
psql -h localhost -U app -d app -c "
  SELECT pid, now() - pg_stat_activity.query_start AS duration, query
  FROM pg_stat_activity
  WHERE (now() - pg_stat_activity.query_start) > interval '30 seconds'
    AND state != 'idle'
  ORDER BY duration DESC;
"

# ✅ GOOD — Prometheus export of database metrics for monitoring
# Add to application /metrics endpoint or use pg_exporter
# Metrics exposed:
# pg_stat_activity_count{datname="app"}
# pg_stat_activity_max_tx_duration{datname="app"}
# pg_up{instance="localhost:5432"}

# Grafana query for database connection pool monitoring:
# avg(rate(pg_stat_activity_count{datname="app"}[5m])) by (state)
# max(pg_settings_max_connections{instance="localhost:5432"})

Pattern 6: External Service Health Checks with Circuit Breaker

Implement circuit breaker patterns for external service dependencies to prevent cascading failures. Use CLI-based health checking with retry and backoff logic.

# ❌ BAD — no circuit breaker, repeated failures overwhelm external service
curl -s https://api.example.com/data

# ✅ GOOD — health check with circuit breaker pattern via curl + shell state
#!/bin/bash
# circuit_breaker_check.sh — Circuit breaker for external service health checks

API_URL="https://api.example.com"
CIRCUIT_BREAKER_STATE_FILE="/tmp/circuit_breaker_state.json"
FAILURE_THRESHOLD=5
RECOVERY_TIMEOUT=30
MAX_RETRIES=3
BASE_DELAY=1

# Initialize circuit breaker state file if it doesn't exist
init_state() {
    if [[ ! -f "$CIRCUIT_BREAKER_STATE_FILE" ]]; then
        echo '{"state":"closed","failure_count":0,"last_failure":0}' > "$CIRCUIT_BREAKER_STATE_FILE"
    fi
}

# Read current circuit breaker state
get_state() {
    jq -r '.state' "$CIRCUIT_BREAKER_STATE_FILE"
}

# Check if request is allowed based on circuit state
is_allowed() {
    local state
    state=$(get_state)
    
    case "$state" in
        closed)
            return 0  # Allow
            ;;
        open)
            local last_failure
            last_failure=$(jq -r '.last_failure' "$CIRCUIT_BREAKER_STATE_FILE")
            local now
            now=$(date +%s)
            local elapsed=$((now - last_failure))
            
            if (( elapsed >= RECOVERY_TIMEOUT )); then
                # Transition to half-open
                echo '{"state":"half_open","failure_count":0,"last_failure":0}' > "$CIRCUIT_BREAKER_STATE_FILE"
                return 0
            fi
            echo "❌ Circuit breaker OPEN — rejecting request"
            return 1
            ;;
        half_open)
            return 0  # Allow limited requests to test recovery
            ;;
    esac
}

# Record success
record_success() {
    local current_state
    current_state=$(get_state)
    
    if [[ "$current_state" == "half_open" ]]; then
        # Successfully recovered — close circuit
        echo '{"state":"closed","failure_count":0,"last_failure":0}' > "$CIRCUIT_BREAKER_STATE_FILE"
        echo "✅ Circuit breaker CLOSED — service recovered"
    else
        # Reset failure count
        local current
        current=$(cat "$CIRCUIT_BREAKER_STATE_FILE")
        echo "$current" | jq '.failure_count = 0' > "$CIRCUIT_BREAKER_STATE_FILE"
    fi
}

# Record failure
record_failure() {
    local current
    current=$(cat "$CIRCUIT_BREAKER_STATE_FILE")
    local failure_count
    failure_count=$(echo "$current" | jq '.failure_count')
    failure_count=$((failure_count + 1))
    local now
    now=$(date +%s)
    
    # Update state
    echo "{\"state\":\"$current_state\",\"failure_count\":${failure_count},\"last_failure\":${now}}" > "$CIRCUIT_BREAKER_STATE_FILE"
    
    if [[ "$current_state" == "closed" ]] && (( failure_count >= FAILURE_THRESHOLD )); then
        echo '{"state":"open","failure_count":'"${failure_count}"',"last_failure":'"${now}"'}' > "$CIRCUIT_BREAKER_STATE_FILE"
        echo "❌ Circuit breaker OPEN — failure threshold reached (${failure_count}/${FAILURE_THRESHOLD})"
    fi
}

# Check external service with circuit breaker protection
check_service() {
    local attempt=0
    
    # Check circuit breaker state
    if ! is_allowed; then
        return 1
    fi
    
    while (( attempt < MAX_RETRIES )); do
        local http_code
        http_code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "${API_URL}/data" 2>/dev/null || echo "000")
        
        if [[ "$http_code" -ge 200 && "$http_code" -lt 500 ]]; then
            record_success
            echo "✅ Service healthy (HTTP ${http_code})"
            return 0
        fi
        
        record_failure
        echo "⚠️ Service check failed (HTTP ${http_code}), attempt $((attempt + 1))/${MAX_RETRIES}"
        
        # Exponential backoff with jitter
        local delay=$((BASE_DELAY * (2 ** attempt)))
        local jitter=$((RANDOM % 100))
        sleep "${delay}.${jitter}"
        
        ((attempt++))
    done
    
    return 1
}

# Main
init_state
check_service

Pattern 7: Health Check Testing with Docker Compose

Create comprehensive health check tests that validate all endpoints and dependencies.

# ❌ BAD — basic health check test without dependency validation
curl -f http://localhost:8080/health || exit 1

# ✅ GOOD — comprehensive health check test suite
#!/bin/bash
set -e

HEALTH_ENDPOINT="http://localhost:8080"
EXPECTED_RESPONSE_CODE=200

echo "=== Health Check Test Suite ==="

# Test 1: Basic health endpoint responds
echo "Test 1: Basic health endpoint"
RESPONSE=$(curl -s -w "\n%{http_code}" "$HEALTH_ENDPOINT/health")
HTTP_CODE=$(echo "$RESPONSE" | tail -1)
BODY=$(echo "$RESPONSE" | sed '$d')

if [ "$HTTP_CODE" != "200" ]; then
    echo "FAIL: Expected HTTP 200, got $HTTP_CODE"
    echo "Body: $BODY"
    exit 1
fi
echo "PASS: Health endpoint returns 200"

# Test 2: Liveness endpoint responds
echo "Test 2: Liveness endpoint"
LIVENESS_RESPONSE=$(curl -s -w "\n%{http_code}" "$HEALTH_ENDPOINT/health/live")
LIVENESS_CODE=$(echo "$LIVENESS_RESPONSE" | tail -1)

if [ "$LIVENESS_CODE" != "200" ]; then
    echo "FAIL: Liveness endpoint returned $LIVENESS_CODE"
    exit 1
fi
echo "PASS: Liveness endpoint returns 200"

# Test 3: Readiness endpoint responds
echo "Test 3: Readiness endpoint"
READINESS_RESPONSE=$(curl -s -w "\n%{http_code}" "$HEALTH_ENDPOINT/health/ready")
READINESS_CODE=$(echo "$READINESS_RESPONSE" | tail -1)

if [ "$READINESS_CODE" != "200" ]; then
    echo "FAIL: Readiness endpoint returned $READINESS_CODE"
    exit 1
fi
echo "PASS: Readiness endpoint returns 200"

# Test 4: JSON response structure validation
echo "Test 4: JSON response structure"
if ! echo "$BODY" | python3 -c "import json,sys; d=json.load(sys.stdin); assert 'status' in d" 2>/dev/null; then
    echo "FAIL: Missing 'status' field in response"
    exit 1
fi
echo "PASS: JSON structure valid"

# Test 5: Health endpoint with database dependency
echo "Test 5: Database dependency check"
if ! echo "$BODY" | python3 -c "import json,sys; d=json.load(sys.stdin); assert 'checks' in d and 'database' in d['checks']" 2>/dev/null; then
    echo "FAIL: Database check not included in response"
    exit 1
fi
echo "PASS: Database check included"

echo "=== All Tests Passed ==="

Pattern 8: Kubernetes Health Check Debugging Commands

Debugging health check issues with kubectl commands and log analysis.

Pattern 9: Health Check with livenessProbe/readinessProbe Using kubectl

Using kubectl and shell scripts for Kubernetes-native health monitoring.

# ✅ GOOD — Comprehensive health check using kubectl probe simulation
#!/bin/bash
# k8s_health_check.sh — Health checks matching Kubernetes probe behavior

NAMESPACE="${NAMESPACE:-default}"
POD_NAME="${POD_NAME:-my-app}"

# Simulate liveness probe (exec command check)
check_liveness() {
    echo "=== Liveness Probe (exec) ==="
    
    # Execute command inside container
    kubectl exec -n "$NAMESPACE" "$POD_NAME" -- /bin/sh -c "
        if python -c 'import socket; s=socket.socket(); s.settimeout(2); s.connect((\"localhost\", 8080)); s.close()' 2>/dev/null; then
            echo 'LIVE: Application socket responding'
            exit 0
        else
            echo 'LIVE: Application socket NOT responding'
            exit 1
        fi
    " 2>&1
}

# Simulate readiness probe (HTTP GET check)
check_readiness() {
    echo "=== Readiness Probe (HTTP GET) ==="
    
    # Get pod IP and check readiness endpoint
    POD_IP=$(kubectl get pod -n "$NAMESPACE" "$POD_NAME" -o jsonpath='{.status.podIP}')
    
    curl -sf --max-time 3 "http://${POD_IP}:8080/health/ready" 2>/dev/null
    local exit_code=$?
    
    if [[ $exit_code -eq 0 ]]; then
        echo "READY: Readiness endpoint responding"
        curl -s "http://${POD_IP}:8080/health/ready" | jq '.'
    else
        echo "NOT READY: Readiness endpoint failed (exit code: ${exit_code})"
    fi
}

# Simulate startup probe (extended readiness check)
check_startup() {
    echo "=== Startup Probe ==="
    
    local max_attempts=30
    local attempt=1
    
    while (( attempt <= max_attempts )); do
        if curl -sf --max-time 3 "http://${POD_IP}:8080/health/started" 2>/dev/null; then
            echo "STARTED: Application fully initialized after $((attempt * 5))s"
            return 0
        fi
        echo "  Attempt ${attempt}/${max_attempts} — not yet ready..."
        sleep 5
        ((attempt++))
    done
    
    echo "FAILED: Application failed to start within ${max_attempts} attempts"
    return 1
}

# Run all checks
check_liveness
check_readiness
check_startup

# ✅ GOOD — Kubernetes probe configuration matching health check patterns
apiVersion: v1
kind: Pod
metadata:
  name: app
  namespace: default
spec:
  containers:
  - name: app
    image: my-app:latest
    ports:
    - containerPort: 8080
    # Liveness: quick check, restart if stuck
    livenessProbe:
      httpGet:
        path: /health/live
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 20
      timeoutSeconds: 5
      failureThreshold: 3
    # Readiness: full dependency check, remove from service if failing
    readinessProbe:
      httpGet:
        path: /health/ready
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 3
      failureThreshold: 3
    # Startup: generous timeout for slow initialization
    startupProbe:
      httpGet:
        path: /health/started
        port: 8080
      failureThreshold: 30
      periodSeconds: 5

Pattern 10: Go Health Check with net/http

Go implementation of health checks using standard library.

// ✅ GOOD — Go health check implementation
package main

import (
    "context"
    "encoding/json"
    "log"
    "net/http"
    "os"
    "time"
    
    "github.com/go-redis/redis/v8"
)

type HealthStatus struct {
    Status     string            `json:"status"`
    Checks     map[string]Check  `json:"checks"`
    Timestamp  string            `json:"timestamp"`
}

type Check struct {
    Status  string `json:"status"`
    Error   string `json:"error,omitempty"`
    Latency string `json:"latency,omitempty"`
}

func checkDatabase(ctx context.Context) Check {
    start := time.Now()
    // Database connection logic here
    duration := time.Since(start)
    
    return Check{
        Status:  "healthy",
        Latency: duration.String(),
    }
}

func checkRedis(ctx context.Context, client *redis.Client) Check {
    start := time.Now()
    err := client.Ping(ctx).Err()
    duration := time.Since(start)
    
    if err != nil {
        return Check{
            Status: "unhealthy",
            Error:  err.Error(),
        }
    }
    
    return Check{
        Status:  "healthy",
        Latency: duration.String(),
    }
}

func healthHandler(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
    defer cancel()
    
    checks := make(map[string]Check)
    
    // Run checks
    checks["database"] = checkDatabase(ctx)
    
    redisClient := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    })
    checks["redis"] = checkRedis(ctx, redisClient)
    
    // Determine overall status
    status := "healthy"
    for _, check := range checks {
        if check.Status != "healthy" {
            status = "unhealthy"
            break
        }
    }
    
    if status != "healthy" {
        w.WriteHeader(http.StatusServiceUnavailable)
    }
    
    response := HealthStatus{
        Status:    status,
        Checks:    checks,
        Timestamp: time.Now().UTC().Format(time.RFC3339),
    }
    
    json.NewEncoder(w).Encode(response)
}

func main() {
    http.HandleFunc("/health", healthHandler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Pattern 11: Node.js Health Check with Express

Node.js implementation with Express framework.

// ✅ GOOD — Node.js Express health check
const express = require('express');
const { Pool } = require('pg');
const redis = require('redis');
const app = express();

const pool = new Pool({
    host: process.env.DB_HOST || 'localhost',
    database: process.env.DB_NAME || 'app',
    user: process.env.DB_USER || 'app',
    password: process.env.DB_PASSWORD || 'secret',
    connectionTimeoutMillis: 3000
});

const redisClient = redis.createClient({
    url: process.env.REDIS_URL || 'redis://localhost:6379',
    socket: { timeout: 3000 }
});

async function checkDatabase() {
    try {
        const start = Date.now();
        const client = await pool.connect();
        await client.query('SELECT 1');
        client.release();
        const latency = Date.now() - start;
        
        return { status: 'healthy', latency_ms: latency };
    } catch (error) {
        return { status: 'unhealthy', error: error.message };
    }
}

async function checkRedis() {
    try {
        const start = Date.now();
        await redisClient.ping();
        const latency = Date.now() - start;
        
        return { status: 'healthy', latency_ms: latency };
    } catch (error) {
        return { status: 'unhealthy', error: error.message };
    }
}

app.get('/health', async (req, res) => {
    const checks = {};
    
    try {
        const [dbResult, redisResult] = await Promise.all([
            checkDatabase(),
            checkRedis()
        ]);
        
        checks.database = dbResult;
        checks.redis = redisResult;
        
        const allHealthy = Object.values(checks).every(c => c.status === 'healthy');
        
        res.status(allHealthy ? 200 : 503).json({
            status: allHealthy ? 'healthy' : 'degraded',
            checks,
            timestamp: new Date().toISOString()
        });
    } catch (error) {
        res.status(503).json({
            status: 'unhealthy',
            checks,
            error: error.message
        });
    }
});

// Separate liveness endpoint
app.get('/health/live', (req, res) => {
    res.status(200).json({ status: 'ok' });
});

// Separate readiness endpoint
app.get('/health/ready', async (req, res) => {
    try {
        const results = await Promise.all([
            checkDatabase(),
            checkRedis()
        ]);
        
        const allHealthy = results.every(r => r.status === 'healthy');
        res.status(allHealthy ? 200 : 503).json({
            status: allHealthy ? 'healthy' : 'unhealthy',
            checks: { database: results[0], redis: results[1] }
        });
    } catch (error) {
        res.status(503).json({ status: 'unhealthy', error: error.message });
    }
});

app.listen(8080, () => {
    console.log('Health check server running on port 8080');
});

Pattern 12: Java Spring Boot Health Check

Spring Boot implementation with Actuator.

// ✅ GOOD — Spring Boot Actuator health check
package com.example.health;

import org.springframework.boot.actuate.health.*;
import org.springframework.stereotype.Component;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.RedisTemplate;

import javax.sql.DataSource;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.HashMap;
import java.util.Map;

@Component
public class CustomHealthIndicator implements HealthIndicator {
    
    @Autowired
    private DataSource dataSource;
    
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;
    
    @Override
    public Health health() {
        Map<String, Object> details = new HashMap<>();
        
        Health databaseHealth = checkDatabase(details);
        Health redisHealth = checkRedis(details);
        
        Health.Builder builder = Health.status(databaseHealth.getStatus().getCode());
        if (databaseHealth.getStatus() != Status.UP || redisHealth.getStatus() != Status.UP) {
            builder = Health.down();
        }
        
        return builder
            .withDetails(details)
            .withDetail("timestamp", System.currentTimeMillis())
            .build();
    }
    
    private Health checkDatabase(Map<String, Object> details) {
        try {
            long start = System.currentTimeMillis();
            try (Connection conn = dataSource.getConnection()) {
                if (conn.isValid(3)) {
                    long latency = System.currentTimeMillis() - start;
                    details.put("database", Map.of(
                        "status", "healthy",
                        "latency_ms", latency
                    ));
                    return Health.up().build();
                }
            }
        } catch (SQLException e) {
            details.put("database", Map.of(
                "status", "unhealthy",
                "error", e.getMessage()
            ));
            return Health.down(e).build();
        }
        
        details.put("database", Map.of("status", "unhealthy", "error", "Connection invalid"));
        return Health.down().withDetail("error", "Database connection invalid").build();
    }
    
    private Health checkRedis(Map<String, Object> details) {
        try {
            long start = System.currentTimeMillis();
            String ping = redisTemplate.execute(RedisConnection::ping);
            long latency = System.currentTimeMillis() - start;
            
            if ("PONG".equals(ping)) {
                details.put("redis", Map.of(
                    "status", "healthy",
                    "latency_ms", latency
                ));
                return Health.up().build();
            }
        } catch (Exception e) {
            details.put("redis", Map.of(
                "status", "unhealthy",
                "error", e.getMessage()
            ));
            return Health.down(e).build();
        }
        
        details.put("redis", Map.of("status", "unhealthy", "error", "Redis ping failed"));
        return Health.down().withDetail("error", "Redis ping failed").build();
    }
}

Pattern 13: Health Check Metrics with Prometheus

Exporting health check metrics for monitoring. Use promtool and Prometheus queries for metric validation.

# ✅ GOOD — Health check metrics via Prometheus query interface
# Query database health check duration
curl -s "http://prometheus:9090/api/v1/query" \
  --data-urlencode 'query=health_check_duration_seconds{check_name="database"}' | jq '.data.result[] | {metric, value}'

# Query unhealthy health check count
curl -s "http://prometheus:9090/api/v1/query" \
  --data-urlencode 'query=health_check_total{status="unhealthy"}' | jq '.data.result[] | {metric, value}'

# Check live health gauge (1 = healthy, 0 = unhealthy)
curl -s "http://prometheus:9090/api/v1/query" \
  --data-urlencode 'query=health_check_live' | jq '.data.result[]'

# Alert rule for unhealthy checks (Prometheus rule file)
cat << 'EOF' > /etc/prometheus/rules/health_alerts.yaml
groups:
  - name: health-check-alerts
    rules:
      - alert: HealthCheckUnhealthy
        expr: health_check_total{status="unhealthy"} > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Health check failing for {{ $labels.check_name }}"
          description: "Check {{ $labels.check_name }} has been unhealthy for >5 minutes"
EOF

# Validate alert rules with promtool
promtool check rules /etc/prometheus/rules/health_alerts.yaml

# Check metrics endpoint directly from the application
curl -s http://localhost:9090/metrics | grep "health_check_" | head -20

# ✅ GOOD — Prometheus scrape configuration for health checks
# prometheus-scrape-config.yaml
- job_name: 'health-checks'
  metrics_path: '/metrics'
  scrape_interval: 15s
  scrape_timeout: 5s
  static_configs:
    - targets: ['app:9090']
      labels:
        environment: 'production'

Pattern 14: Health Check with Retry Logic

Implementing retry logic with exponential backoff and jitter for transient failures.

# ✅ GOOD — Health check with exponential backoff retry in bash
#!/bin/bash
# health_check_retry.sh — Retry logic with exponential backoff

HEALTH_ENDPOINT="http://localhost:8080/health"
MAX_RETRIES=3
BASE_DELAY=1    # seconds
MAX_DELAY=10    # seconds

check_with_retry() {
    local endpoint=$1
    local attempt=0
    
    while (( attempt < MAX_RETRIES )); do
        local http_code
        http_code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$endpoint" 2>/dev/null || echo "000")
        
        if [[ "$http_code" -ge 200 && "$http_code" -lt 400 ]]; then
            echo "✅ Health check passed (HTTP ${http_code}) after ${attempt} retry attempt(s)"
            return 0
        fi
        
        echo "⚠️ Health check failed (HTTP ${http_code}), attempt $((attempt + 1))/${MAX_RETRIES}"
        
        ((attempt++))
        
        if (( attempt < MAX_RETRIES )); then
            # Exponential backoff with jitter
            local delay=$((BASE_DELAY * (2 ** (attempt - 1))))
            # Cap at max_delay
            if (( delay > MAX_DELAY )); then
                delay=$MAX_DELAY
            fi
            # Add jitter (0 to delay/10)
            local jitter=$((RANDOM % (delay / 10 + 1)))
            local total_delay=$((delay + jitter))
            
            echo "  Backing off for ${total_delay}s before retry..."
            sleep "$total_delay"
        fi
    done
    
    echo "❌ Health check failed after ${MAX_RETRIES} attempts"
    return 1
}

# Check multiple endpoints with independent retry
check_with_retry "$HEALTH_ENDPOINT/health/live"
check_with_retry "$HEALTH_ENDPOINT/health/ready"
check_with_retry "$HEALTH_ENDPOINT/health"

# ✅ GOOD — Retry logic using the `retry` utility (GNU coreutils)
# Simple retry with fixed delay
retry --delay 2 --attempts 5 curl -sf http://localhost:8080/health || echo "Failed"

# Retry with exponential backoff using wait (macOS/BSD)
# or sleep with exponential calculation (Linux)
for i in $(seq 1 5); do
    if curl -sf http://localhost:8080/health > /dev/null 2>&1; then
        echo "✅ Service healthy on attempt $i"
        exit 0
    fi
    echo "⚠️ Attempt $i failed, retrying..."
    sleep $((2 ** i))
done
echo "❌ All retries exhausted"

Pattern 15: Health Check in Docker Container

Docker container health check configuration.

# ✅ GOOD — Docker health check configuration
FROM python:3.11-slim

WORKDIR /app

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application
COPY app.py .

# Health check with curl
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8080/health/live || exit 1

EXPOSE 8080

CMD ["python", "app.py"]

# Alternative: Docker Compose health check
version: '3.8'

services:
  app:
    image: my-app:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health/live"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    ports:
      - "8080:8080"
  
  db:
    image: postgres:15
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
    environment:
      POSTGRES_PASSWORD: secret

Pattern 16: Health Check with Graceful Degradation

Implementing graceful degradation for non-critical dependencies using classification tiers.

# ✅ GOOD — Health check with dependency tier classification
#!/bin/bash
# graceful_degradation_check.sh — Tiered health checks

declare -A DEPENDENCY_TIERS
DEPENDENCY_TIERS[database]="critical"
DEPENDENCY_TIERS[cache]="optional"
DEPENDENCY_TIERS[metrics]="non_critical"
DEPENDENCY_TIERS[external_api]="non_critical"

declare -A CHECK_RESULTS

check_database() {
    if curl -sf --max-time 3 http://localhost:8080/db/ping > /dev/null 2>&1; then
        CHECK_RESULTS[database]="healthy"
    else
        CHECK_RESULTS[database]="unhealthy"
    fi
}

check_cache() {
    if redis-cli -h localhost ping 2>/dev/null | grep -q PONG; then
        CHECK_RESULTS[cache]="healthy"
    else
        CHECK_RESULTS[cache]="unhealthy"
    fi
}

check_metrics() {
    if curl -sf --max-time 3 http://localhost:9090/metrics > /dev/null 2>&1; then
        CHECK_RESULTS[metrics]="healthy"
    else
        CHECK_RESULTS[metrics]="unhealthy"
    fi
}

check_external_api() {
    if curl -sf --max-time 3 https://api.example.com/status > /dev/null 2>&1; then
        CHECK_RESULTS[external_api]="healthy"
    else
        CHECK_RESULTS[external_api]="unhealthy"
    fi
}

# Run all checks
check_database
check_cache
check_metrics
check_external_api

# Determine overall status with graceful degradation
has_critical_failure=false
has_optional_failure=false

for check_name in "${!CHECK_RESULTS[@]}"; do
    tier="${DEPENDENCY_TIERS[$check_name]}"
    result="${CHECK_RESULTS[$check_name]}"
    
    if [[ "$result" == "unhealthy" ]]; then
        echo "⚠️ $check_name ($tier): unhealthy"
        if [[ "$tier" == "critical" ]]; then
            has_critical_failure=true
        elif [[ "$tier" == "optional" ]]; then
            has_optional_failure=true
        fi
    else
        echo "✅ $check_name ($tier): healthy"
    fi
done

# Determine HTTP response code
if $has_critical_failure; then
    HTTP_STATUS=503
    OVERALL="unhealthy"
elif $has_optional_failure; then
    HTTP_STATUS=200  # Still serving, but degraded
    OVERALL="degraded"
else
    HTTP_STATUS=200
    OVERALL="healthy"
fi

echo ""
echo "=== Overall Status: $OVERALL (HTTP $HTTP_STATUS) ==="
echo "Critical failures: $has_critical_failure"
echo "Optional failures: $has_optional_failure"

# ✅ GOOD — Kubernetes annotations for graceful degradation
# Service configuration with health check awareness
apiVersion: v1
kind: Service
metadata:
  name: my-app
  annotations:
    # Health check classification for service mesh
    healthcheck.tier: "critical"
    healthcheck.graceful-degradation: "true"
    # Service mesh will route around degraded pods
    sidecar.istio.io/inject: "true"
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080

Pattern 17: Health Check with Rate Limiting

Implementing rate limiting on health check endpoints.

# ✅ GOOD — Rate limiting with curl and NGINX configuration
# NGINX rate limiting for health endpoints
cat << 'EOF' > /etc/nginx/conf.d/health-rate-limit.conf
limit_req_zone $binary_remote_addr zone:health_limit:10m rate=10r/m;
limit_req_zone $binary_remote_addr zone:liveness_limit:10m rate=100r/m;
limit_req_zone $binary_remote_addr zone:ready_limit:10m rate=30r/m;
limit_req_zone $binary_remote_addr zone:metrics_limit:10m rate=5r/m;

server {
    location /health { limit_req zone=health_limit burst=5 nodelay; proxy_pass http://app:8080/health; }
    location /health/live { limit_req zone=liveness_limit burst=20 nodelay; proxy_pass http://app:8080/health/live; }
    location /health/ready { limit_req zone=ready_limit burst=10 nodelay; proxy_pass http://app:8080/health/ready; }
    location /health/metrics { limit_req zone=metrics_limit burst=2 nodelay; proxy_pass http://app:8080/health/metrics; }
}
EOF
nginx -t && nginx -s reload

# ✅ GOOD — Test rate limiting with curl
for i in $(seq 1 15); do
    HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" http://localhost/health)
    echo "Request $i: HTTP $HTTP_CODE"
    [[ "$HTTP_CODE" == "429" ]] && { echo "Rate limited"; break; }
done

Pattern 18: Health Check with Cache

Caching health check results to reduce load on dependencies. Use Redis/TTL or file-based caching.

# ✅ GOOD — Health check with TTL-based caching
#!/bin/bash
# cached_health_check.sh — Health checks with result caching

CACHE_DIR="/tmp/health-check-cache"
CACHE_TTL=5  # seconds
HEALTH_ENDPOINT="http://localhost:8080/health"

# Initialize cache directory
mkdir -p "$CACHE_DIR"

# Check if cached result is still valid
is_cache_valid() {
    local check_name=$1
    local cache_file="${CACHE_DIR}/${check_name}"
    
    if [[ -f "$cache_file" ]]; then
        local cache_time
        cache_time=$(stat -c %Y "$cache_file" 2>/dev/null || stat -f %m "$cache_file" 2>/dev/null)
        local now
        now=$(date +%s)
        local age=$((now - cache_time))
        
        if (( age < CACHE_TTL )); then
            return 0  # Cache valid
        fi
    fi
    return 1  # Cache expired or missing
}

# Get cached result
get_cached_result() {
    local check_name=$1
    local cache_file="${CACHE_DIR}/${check_name}"
    
    if is_cache_valid "$check_name"; then
        cat "$cache_file"
        return 0
    fi
    return 1
}

# Save result to cache
save_to_cache() {
    local check_name=$1
    local result=$2
    local cache_file="${CACHE_DIR}/${check_name}"
    
    echo "$result" > "$cache_file"
}

# Perform health check with caching
check_health() {
    local check_name=$1
    
    # Try cached result first
    if cached_result=$(get_cached_result "$check_name"); then
        echo "✅ Cached result ($check_name): $cached_result"
        return 0
    fi
    
    # Perform fresh check
    local http_code
    http_code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$HEALTH_ENDPOINT" 2>/dev/null || echo "000")
    
    local result
    if [[ "$http_code" -ge 200 && "$http_code" -lt 400 ]]; then
        result="healthy"
        echo "✅ Fresh check ($check_name): healthy (HTTP ${http_code})"
    else
        result="unhealthy"
        echo "❌ Fresh check ($check_name): unhealthy (HTTP ${http_code})"
    fi
    
    # Save to cache
    save_to_cache "$check_name" "$result"
}

# Run checks
check_health "database"
check_health "cache"
check_health "external_api"

# ✅ GOOD — Redis-based health check cache configuration
# Cache health check results in Redis for distributed systems
apiVersion: v1
kind: ConfigMap
metadata:
  name: health-check-cache-config
data:
  # TTL for health check cache in seconds
  cache

…(truncated)