Graceful Shutdown

Graceful Shutdown

Process signal handling, connection draining, and clean resource teardown for NodeJS-Starter-V1.

Metadata

Description

Codifies the shutdown lifecycle for both the FastAPI backend and Next.js frontend in NodeJS-Starter-V1. Covers OS signal handling (SIGTERM, SIGINT), connection pool draining, active request completion, WebSocket teardown, scheduled timer cancellation, subagent cleanup, and Docker container stop_grace_period integration.

When to Apply

Positive Triggers

• Implementing or modifying the FastAPI lifespan context manager
• Adding new persistent resources (database pools, WebSocket connections, MCP clients)
• Writing background workers, scheduled tasks, or long-running agent operations
• Configuring Docker Compose stop_grace_period or Kubernetes terminationGracePeriodSeconds
• Integrating health-check readiness probes with shutdown awareness
• Handling SIGTERM or SIGINT in Python or Node.js processes

Negative Triggers

• Pure stateless API endpoints with no persistent connections
• Static asset serving or build-time generation
• Client-side React components (use useEffect cleanup instead — not this skill)
• One-shot CLI scripts that exit after completion

Core Principles

The Three Laws of Graceful Shutdown

1. Signal → Stop Accepting → Drain → Teardown: Always follow this four-phase sequence. Never skip phases.
2. Bounded Grace Period: Every shutdown has a hard deadline. If draining exceeds the timeout, force-terminate.
3. Health Probe Awareness: Mark readiness as false immediately on signal receipt, before draining begins.

Phase 1: Signal Handling

Python (FastAPI / Uvicorn)

Uvicorn handles SIGTERM and SIGINT natively. The FastAPI lifespan context manager is the primary integration point:

from contextlib import asynccontextmanager
from typing import AsyncGenerator

from fastapi import FastAPI

from src.utils import get_logger

logger = get_logger(__name__)

# Shared shutdown event for coordinating background tasks
shutdown_event = asyncio.Event()

@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
    """Application lifespan — startup and shutdown hooks."""
    logger.info("Starting application")
    # Startup: initialise resources
    yield
    # Shutdown: teardown resources (runs on SIGTERM/SIGINT)
    logger.info("Shutdown signal received — beginning graceful shutdown")
    shutdown_event.set()
    await drain_and_teardown()
    logger.info("Graceful shutdown complete")

Project Reference: apps/backend/src/api/main.py:20-26 — current lifespan (startup logging only, shutdown is a log-only stub).

TypeScript (Next.js / Node.js)

For standalone Node.js processes or custom Next.js servers:

const shutdownController = new AbortController();

function registerShutdownHandlers(cleanup: () => Promise<void>): void {
  const signals: NodeJS.Signals[] = ['SIGTERM', 'SIGINT'];
  let shuttingDown = false;

  for (const signal of signals) {
    process.on(signal, async () => {
      if (shuttingDown) return; // Prevent double-shutdown
      shuttingDown = true;
      console.log(`Received ${signal} — beginning graceful shutdown`);
      shutdownController.abort();

      const timeout = setTimeout(() => {
        console.error('Shutdown timed out — forcing exit');
        process.exit(1);
      }, 30_000); // 30s hard deadline

      try {
        await cleanup();
      } finally {
        clearTimeout(timeout);
        process.exit(0);
      }
    });
  }
}

Phase 2: Stop Accepting New Work

Backend — Reject New Requests

Once a shutdown signal is received, the backend must stop accepting new requests while allowing in-flight requests to complete:

from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import JSONResponse

class ShutdownAwareMiddleware(BaseHTTPMiddleware):
    """Returns 503 for new requests during shutdown."""

    async def dispatch(self, request: Request, call_next):
        if shutdown_event.is_set():
            # Allow health probes through for orchestrator awareness
            if request.url.path in ("/health", "/ready"):
                return await call_next(request)
            return JSONResponse(
                status_code=503,
                content={
                    "detail": "Service is shutting down",
                    "retry_after": 5,
                },
                headers={"Retry-After": "5"},
            )
        return await call_next(request)

Frontend — Stop Scheduled Timers

Cancel all setInterval and setTimeout handles:

// Pattern from ScheduledAuditRunner
stopAll(): void {
  for (const scheduleId of this.timers.keys()) {
    this.stopSchedule(scheduleId);
  }
}

Project Reference: apps/web/lib/audit/scheduled-audit-runner.ts:526-530 — stopAll() clears all interval timers.

Phase 3: Drain Active Connections

Database Connection Pool

Dispose the SQLAlchemy async engine to drain active connections:

from src.config.database import async_engine, sync_engine

async def drain_database() -> None:
    """Dispose database connection pools."""
    logger.info("Draining database connections")
    await async_engine.dispose()
    sync_engine.dispose()
    logger.info("Database connections drained")

Project Reference: apps/backend/src/config/database.py:60-66 — async_engine with pool_size=5, max_overflow=10. The dispose() method waits for checked-out connections to return before closing the pool.

Subagent Manager

Cancel running subagents and clean up tracking state:

async def drain_subagents(manager: SubagentManager) -> None:
    """Cancel active subagents and clean up."""
    active_count = await manager.get_active_count()
    if active_count > 0:
        logger.info("Cancelling active subagents", count=active_count)
        for subtask_id, tracking in manager._active_subagents.items():
            if tracking["status"] == SubagentStatus.RUNNING:
                await manager.cancel_subagent(subtask_id)
    await manager.cleanup()
    logger.info("Subagent cleanup complete")

Project Reference: apps/backend/src/agents/subagent_manager.py:485-507 — cancel_subagent() marks agents as CANCELLED; cleanup() at line 520 removes completed/failed/cancelled entries.

MCP Client Connections

Disconnect from all MCP servers:

async def drain_mcp_connections(client: MCPClient) -> None:
    """Disconnect from all MCP servers."""
    status = await client.get_connection_status()
    connected = sum(1 for v in status.values() if v)
    if connected > 0:
        logger.info("Disconnecting MCP servers", count=connected)
        await client.cleanup()

Project Reference: apps/backend/src/tools/mcp_client.py:295-302 — cleanup() iterates all connections and calls disconnect_from_server().

WebSocket / Yjs Provider

Tear down real-time collaboration connections:

function destroyCollaboration(provider: YjsProvider): void {
  provider.destroy();
  // destroy() does:
  //   awareness.off('change', updateCollaborators)
  //   wsProvider.disconnect()
  //   ydoc.destroy()
}

Project Reference: apps/web/lib/collaboration/yjs-provider.ts:241-245 — destroy() removes listeners, disconnects WebSocket, and destroys the Yjs document.

Phase 4: Teardown Orchestration

Combined Shutdown Function

Orchestrate all drain operations with a bounded timeout:

import asyncio

SHUTDOWN_TIMEOUT_SECONDS = 30

async def drain_and_teardown() -> None:
    """Orchestrate all shutdown operations with timeout."""
    try:
        await asyncio.wait_for(
            _drain_all(),
            timeout=SHUTDOWN_TIMEOUT_SECONDS,
        )
    except TimeoutError:
        logger.error(
            "Shutdown timed out — some resources may not have been released",
            timeout=SHUTDOWN_TIMEOUT_SECONDS,
        )

async def _drain_all() -> None:
    """Drain all resources in dependency order."""
    # Phase 1: Stop accepting (handled by middleware)

    # Phase 2: Drain active work (parallel — independent resources)
    await asyncio.gather(
        drain_subagents(subagent_manager),
        drain_mcp_connections(mcp_client),
        return_exceptions=True,
    )

    # Phase 3: Drain connection pools (after active work completes)
    await drain_database()

    logger.info("All resources released")

Dependency Order

Resources must be drained in reverse-initialisation order:

Signal Received
  ├─ Set shutdown_event (stops new requests)
  ├─ Mark readiness probe as NOT READY
  ├─ Wait for in-flight requests (bounded)
  ├─ Cancel subagents (parallel)
  ├─ Disconnect MCP clients (parallel)
  ├─ Disconnect WebSocket providers
  ├─ Dispose database pools
  └─ Exit process

Rule: Resources that depend on the database (subagents, MCP tools) must drain before the database pool is disposed.

Docker Integration

stop_grace_period

Configure Docker Compose to allow sufficient time for draining:

services:
  backend:
    # ...
    stop_grace_period: 35s  # > SHUTDOWN_TIMEOUT_SECONDS (30s)
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:8000/ready || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 10s

Rule: stop_grace_period must be greater than the application's internal shutdown timeout. Docker sends SIGTERM, waits for stop_grace_period, then sends SIGKILL.

Kubernetes

spec:
  terminationGracePeriodSeconds: 35
  containers:
    - name: backend
      lifecycle:
        preStop:
          exec:
            command: ["/bin/sh", "-c", "sleep 5"]
      # preStop delay allows load balancer to deregister
      # before the app begins draining

Project Reference: docker-compose.yml:15-19 — PostgreSQL healthcheck uses pg_isready. The backend service (commented out) should add stop_grace_period when uncommented.

Health Probe Integration

Readiness Probe During Shutdown

The readiness endpoint must return 503 immediately after shutdown signal, even while draining:

@router.get("/ready")
async def readiness_check() -> JSONResponse:
    """Readiness probe — returns 503 during shutdown."""
    if shutdown_event.is_set():
        return JSONResponse(
            status_code=503,
            content={
                "status": "shutting_down",
                "timestamp": datetime.now().isoformat(),
            },
        )
    return JSONResponse(
        content={
            "status": "ready",
            "timestamp": datetime.now().isoformat(),
        },
    )

Liveness Probe During Shutdown

The liveness probe should continue returning 200 during shutdown to prevent the orchestrator from killing the process prematurely:

@router.get("/health")
async def health_check() -> dict[str, str]:
    """Liveness probe — always returns 200 (even during shutdown)."""
    return {
        "status": "healthy",
        "timestamp": datetime.now().isoformat(),
        "version": "0.1.0",
    }

Project Reference: apps/backend/src/api/routes/health.py:1-28 — current /health and /ready endpoints are not shutdown-aware.

Frontend Cleanup Patterns

React useEffect Cleanup

For components with subscriptions, timers, or WebSocket connections:

useEffect(() => {
  const provider = createYjsProvider(config);

  return () => {
    // Cleanup runs on unmount or dependency change
    provider.destroy();
  };
}, [config.workflowId]);

AbortController for Fetch Requests

Cancel in-flight fetch requests on component unmount or shutdown:

useEffect(() => {
  const controller = new AbortController();

  async function fetchData() {
    try {
      const response = await fetch('/api/data', {
        signal: controller.signal,
      });
      // handle response
    } catch (error) {
      if (error instanceof DOMException && error.name === 'AbortError') {
        return; // Expected during cleanup
      }
      throw error;
    }
  }

  fetchData();
  return () => controller.abort();
}, []);

Logging During Shutdown

All shutdown operations must log structured events for post-mortem analysis:

# Start of shutdown
logger.info("Shutdown signal received", signal="SIGTERM")

# Each drain phase
logger.info("Draining database connections", pool_size=5, active=2)
logger.info("Database connections drained", duration_ms=150)

# Completion or timeout
logger.info("Graceful shutdown complete", total_duration_ms=1200)
# or
logger.error("Shutdown timed out", timeout=30, remaining_connections=1)

Anti-Patterns

Checklist

Before merging code that touches shutdown:

• shutdown_event is set on signal receipt
• Readiness probe returns 503 during shutdown
• Liveness probe returns 200 during shutdown
• New requests rejected with 503 + Retry-After header
• All persistent resources have drain/cleanup methods
• Drain order respects dependency graph (dependents before dependencies)
• Hard timeout prevents infinite drain
• Docker stop_grace_period exceeds application timeout
• Structured log events at each shutdown phase
• Double-signal guard prevents re-entrant shutdown

Response Format

When applying this skill, structure implementation as:

### Shutdown Implementation

**Signal Handler**: [lifespan / process.on / both]
**Resources to Drain**: [list of resources]
**Drain Order**: [dependency-ordered sequence]
**Timeout**: [seconds]
**Health Probe Updates**: [readiness 503 / liveness unchanged]