Health Check

Health Check - Liveness, Readiness & Dependency Probes

Codifies the project's three-tier health probe architecture (shallow liveness, readiness, deep dependency checks), Docker container healthchecks, cron-based periodic monitoring, route discovery verification, and the comprehensive system health script. Patterns align with Kubernetes probe conventions even when running outside K8s.

Description

Codifies liveness, readiness, and deep dependency health endpoints for NodeJS-Starter-V1's Next.js and FastAPI services, covering three-tier probe architecture, Docker healthchecks, cron-based monitoring, route discovery verification, and the system health script.

When to Apply

Positive Triggers

• Adding new health check endpoints or probes
• Integrating new dependencies that need health verification
• Configuring Docker healthchecks for containers
• Setting up periodic health monitoring via cron
• Implementing startup, liveness, or readiness probes
• Adding service dependency checks to existing endpoints
• User mentions: "health check", "liveness", "readiness", "probe", "heartbeat", "service health", "dependency check"

Negative Triggers

• Collecting application metrics (use metrics-collector instead)
• Adding structured log statements (use structured-logging instead)
• Designing dashboard UI for health status (use dashboard-patterns instead)
• Implementing graceful shutdown (use graceful-shutdown when available)

Core Directives

The Three Laws of Health Checks

1. Three tiers, not one: Separate liveness (am I alive?), readiness (can I serve traffic?), and deep (are all dependencies healthy?). Never combine them.
2. Parallel dependency checks: Check all dependencies concurrently via Promise.all or asyncio.gather. Never check sequentially — a slow database should not delay the Redis check.
3. 503 for unhealthy: Return HTTP 200 for healthy/degraded, HTTP 503 for unhealthy. Load balancers and orchestrators use status codes, not response bodies.

Existing Project Infrastructure

Backend (FastAPI)

Frontend (Next.js)

Docker

System Script

scripts/health-check.ps1 — 6-phase comprehensive health check (prerequisites, database, backend, frontend, integration, summary) with exit code 0 (healthy) or 1 (unhealthy).

Health Status Model

All health endpoints use a three-state status:

Aggregation Rule

if any dependency is unhealthy → overall = unhealthy (503)
else if any dependency is degraded → overall = degraded (200)
else → overall = healthy (200)

Probe Patterns

Tier 1: Liveness (Shallow)

Returns immediately with minimal computation. Used by load balancers and orchestrators to confirm the process is alive.

Backend (/health):

@router.get("/health")
async def health_check() -> dict[str, str]:
    return {
        "status": "healthy",
        "timestamp": datetime.now().isoformat(),
        "version": "0.1.0",
    }

Frontend (/api/health):

interface HealthResponse {
  status: "healthy" | "degraded" | "unhealthy";
  timestamp: string;
  version: string;
  uptime: number;
  environment: string;
}

Rules: No database calls, no external service checks, no computation. Must respond in < 50ms.

Tier 2: Readiness

Confirms the service can accept and process requests. Checks that critical dependencies are reachable.

Backend (/ready):

@router.get("/ready")
async def readiness_check() -> dict[str, str]:
    # Check database connectivity
    # Check Redis connectivity
    # Check AI provider availability
    return {"status": "ready", "timestamp": datetime.now().isoformat()}

Rules: Check only fast, critical dependencies (database, cache). Timeout each check at 2–5 seconds. Do not check optional or slow services.

Tier 3: Deep Dependency Check

Checks all dependencies in parallel with latency measurement. Used for debugging and monitoring dashboards, not for load balancer probes (too slow).

Frontend (/api/health/deep):

interface DependencyCheck {
  name: string;
  status: "healthy" | "degraded" | "unhealthy" | "unchecked";
  latency_ms: number | null;
  error: string | null;
  last_checked: string;
}

Each dependency checker follows this pattern:

1. Record start time
2. Attempt operation with timeout (AbortSignal.timeout(5000))
3. Measure latency (Date.now() - start)
4. Classify: healthy (success), degraded (slow or partial), unhealthy (error/timeout)

Checks run in parallel via Promise.all:

const [database, backend, verification] = await Promise.all([
  checkDatabase(),
  checkBackend(),
  checkVerificationSystem(),
]);

The summary aggregates results:

const summary = {
  total_checks: checks.length,
  passed: checks.filter(c => c.status === "healthy").length,
  failed: checks.filter(c => c.status === "unhealthy").length,
  degraded: checks.filter(c => c.status === "degraded").length,
};

Docker Healthcheck Pattern

Docker Compose healthchecks use CMD-SHELL with service-native commands:

services:
  postgres:
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U starter_user -d starter_db"]
      interval: 10s
      timeout: 5s
      retries: 5
  redis:
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

For application containers, use curl or wget against the liveness endpoint:

backend:
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
    interval: 30s
    timeout: 10s
    retries: 3
    start_period: 40s

start_period gives the application time to initialise before healthchecks begin. Use depends_on with condition: service_healthy to sequence container startup.

Cron-Based Monitoring

The project's /api/cron/health-check runs every 5 minutes, pings the backend, and logs results. Secured with CRON_SECRET bearer token.

Pattern for adding new periodic checks:

export async function GET(request: Request) {
  // 1. Verify CRON_SECRET
  const authHeader = request.headers.get("authorization");
  if (authHeader !== `Bearer ${process.env.CRON_SECRET}`) {
    return new NextResponse("Unauthorized", { status: 401 });
  }
  // 2. Run checks with latency measurement
  const start = Date.now();
  const response = await fetch(`${backendUrl}/health`, {
    signal: AbortSignal.timeout(5000),
  });
  const latency = Date.now() - start;
  // 3. Log results
  logger.info("Health check cron", { backend: response.ok, latency });
  // 4. Alert if unhealthy
  if (!response.ok) { logger.error("Backend unhealthy"); }
  // 5. Return results
  return NextResponse.json({ status: response.ok ? "healthy" : "unhealthy" });
}

Route Health Verification

The /api/health/routes endpoint discovers all API routes by scanning the filesystem and optionally verifies each GET endpoint:

1. Discovery: Recursively scan app/api/ for route.ts files
2. Method detection: Parse file content for exported HTTP methods (GET, POST, PUT, PATCH, DELETE)
3. Verification (optional ?verify=true): Send GET request to each endpoint with 5-second timeout
4. Status: verified (200 OK), error (non-200 or timeout), unverified (not tested)

Adding a New Dependency Check

When integrating a new service, add a checker following this template:

async function checkNewService(): Promise<DependencyCheck> {
  const start = Date.now();
  const result: DependencyCheck = {
    name: "service_name",
    status: "unchecked",
    latency_ms: null,
    error: null,
    last_checked: new Date().toISOString(),
  };
  try {
    // Service-specific check (e.g., ping, SELECT 1, PING)
    const response = await fetch(serviceUrl, {
      signal: AbortSignal.timeout(5000),
    });
    result.latency_ms = Date.now() - start;
    result.status = response.ok ? "healthy" : "degraded";
    if (!response.ok) result.error = `HTTP ${response.status}`;
  } catch (e) {
    result.latency_ms = Date.now() - start;
    result.status = "unhealthy";
    result.error = e instanceof Error ? e.message : "Unknown error";
  }
  return result;
}

Then add it to the Promise.all array in the deep health endpoint.

Anti-Patterns

Checklist for New Health Endpoints

Structure

• Three-tier separation (liveness, readiness, deep)
• healthy / degraded / unhealthy status values
• HTTP 200 for healthy/degraded, 503 for unhealthy
• Response includes timestamp and version

Dependencies

• Parallel checking via Promise.all or asyncio.gather
• 5-second timeout per dependency check
• Latency measurement per dependency
• Graceful handling of missing environment variables

Docker

• Container healthcheck using service-native command
• interval, timeout, retries configured
• start_period covers application boot time
• condition: service_healthy for dependent services

Monitoring

• Cron-based periodic checks for production
• CRON_SECRET authentication on cron endpoints
• Health check latency instrumented via metrics-collector
• Failures logged via structured-logging

Response Format

[AGENT_ACTIVATED]: Health Check
[PHASE]: {Design | Implementation | Review}
[STATUS]: {in_progress | complete}

{health check analysis or implementation guidance}

[NEXT_ACTION]: {what to do next}

Integration Points

Metrics Collector

• health_check_duration_ms histogram per dependency
• health_check_status gauge (1=healthy, 0.5=degraded, 0=unhealthy)

Structured Logging

• Info-level health check results (dependency, status, latency)
• Error-level alerts when dependencies become unhealthy

Error Taxonomy

• SYS_HEALTH_DEPENDENCY_UNAVAILABLE (503) — critical dependency unreachable
• SYS_HEALTH_TIMEOUT (504) — dependency check exceeded timeout

Cron Scheduler

• /api/cron/health-check runs every 5 minutes with CRON_SECRET auth
• Results can trigger alerting via notification system

Dashboard Patterns

• StatusPulse component for live dependency status indicators
• DataStrip for health check latency metrics
• Connection status mapped to spectral colours (emerald=healthy, amber=degraded, red=unhealthy)

Australian Localisation (en-AU)

• Spelling: initialise, serialise, analyse, optimise, colour, behaviour
• Date: ISO 8601 in responses; DD/MM/YYYY in dashboard display
• Timezone: AEST/AEDT — timestamps stored as UTC