Production Docker

Production Docker Hardening

This skill transforms demo-quality Docker setups into production-grade container infrastructure. Every recommendation here comes from real incidents: breached containers running as root, 2GB images that take 8 minutes to deploy, secrets leaked into image layers. Follow this guide and none of that happens on your watch.

1. Multi-Stage Builds

Single-stage builds ship compilers, build tools, and source code to production. Multi-stage builds fix this by separating the build environment from the runtime environment.

Python example (builder + distroless):

# syntax=docker/dockerfile:1

FROM python:3.12-slim AS builder
WORKDIR /build
COPY requirements.txt .
RUN pip install --no-cache-dir --no-compile --prefix=/install -r requirements.txt

FROM gcr.io/distroless/python3-debian12
COPY --from=builder /install /usr/local
COPY --chown=65532:65532 ./app /app
WORKDIR /app
USER 65532
ENTRYPOINT ["python", "-m", "app.main"]

Node.js example (builder + slim runtime):

# syntax=docker/dockerfile:1

FROM node:22-slim AS builder
WORKDIR /build
COPY package*.json ./
RUN npm ci --ignore-scripts
COPY . .
RUN npm run build && npm prune --production

FROM node:22-slim
RUN apt-get update && apt-get install -y --no-install-recommends tini && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --from=builder --chown=node:node /build/dist ./dist
COPY --from=builder --chown=node:node /build/node_modules ./node_modules
COPY --from=builder --chown=node:node /build/package.json ./
USER node
ENTRYPOINT ["tini", "--"]
CMD ["node", "dist/index.js"]

Size impact: A typical Python app goes from ~1.2GB (python:3.12 with build deps) to ~120MB (distroless). That is a 90%+ reduction.

Rules:

• The builder stage installs dependencies and compiles code
• The runtime stage copies only the final artifacts
• Never install build tools (gcc, make, git) in the runtime stage
• Use COPY --from=builder to cherry-pick what you need

2. Base Image Selection

Choose the smallest image that supports your runtime. Each option has tradeoffs:

Pinning rules:

• Pin by digest for reproducible builds: FROM python:3.12-slim@sha256:abc123...
• Pin by minor version at minimum: FROM python:3.12-slim, never FROM python:latest
• Never use latest — it is a moving target that breaks builds silently
• Update digests on a schedule (monthly or via Dependabot/Renovate)

Distroless advantages:

• No shell means attackers who get code execution cannot spawn a shell
• No package manager means no installing tools post-exploitation
• Minimal filesystem reduces vulnerability surface to near zero
• CVE scan results are dramatically cleaner

3. Non-Root Execution

Running as root inside a container is the single most common Docker security mistake. If an attacker escapes the container with UID 0, they have root on the host.

Distroless (use the built-in nonroot user):

USER 65532

Debian/Ubuntu (create a dedicated user):

RUN groupadd --gid 10001 appuser && \
    useradd --uid 10001 --gid appuser --shell /bin/false --create-home appuser
USER appuser

Alpine (create a dedicated user):

RUN addgroup -g 10001 -S appuser && \
    adduser -u 10001 -S -G appuser -s /bin/false appuser
USER appuser

Rules:

• Set USER as late as possible in the Dockerfile (after all installs and copies)
• Use --chown on COPY instructions to set correct ownership
• Ensure the app directory and any writable paths are owned by the non-root user
• If the app needs to bind to port 80/443, use a reverse proxy or set CAP_NET_BIND_SERVICE — do not run as root
• Verify at runtime: docker exec <container> whoami should never return root

4. BuildKit Secrets

Secrets (API keys, tokens, private repo credentials) must never appear in image layers. Anyone with docker history or docker save can extract them.

The correct way — BuildKit secret mounts:

# syntax=docker/dockerfile:1
RUN --mount=type=secret,id=pip_index_url \
    PIP_INDEX_URL=$(cat /run/secrets/pip_index_url) \
    pip install --no-cache-dir -r requirements.txt

Build command:

DOCKER_BUILDKIT=1 docker build --secret id=pip_index_url,src=.pip_credentials .

For private git repos:

RUN --mount=type=ssh pip install git+ssh://git@github.com/org/private-repo.git

Build command:

docker build --ssh default .

What NEVER to do:

• ARG MY_SECRET=... — visible in docker history
• ENV MY_SECRET=... — visible in docker inspect
• COPY .env /app/.env — baked into a layer forever
• COPY id_rsa /root/.ssh/ — private key in the image

Rule: If docker history --no-trunc <image> shows any secret value, the image is compromised and must be rebuilt.

5. Layer Optimization

Docker caches layers top-down. When a layer changes, all subsequent layers are invalidated. Order your Dockerfile to maximize cache hits.

Optimal layer order:

# 1. Base image (changes rarely)
FROM python:3.12-slim AS builder

# 2. System dependencies (changes rarely)
RUN apt-get update && apt-get install -y --no-install-recommends \
    libpq-dev \
    && rm -rf /var/lib/apt/lists/*

# 3. App dependencies (changes occasionally)
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --no-compile -r requirements.txt

# 4. Application code (changes frequently)
COPY . .

Rules:

• Copy dependency manifests (requirements.txt, package.json) before application code
• Install dependencies in a separate layer from code copying
• Combine related RUN commands with && to reduce layer count
• Always clean up in the same layer: apt-get install ... && rm -rf /var/lib/apt/lists/*
• Use --no-cache-dir with pip to avoid caching wheels in the layer
• Use --no-compile with pip to skip .pyc generation in the builder

6. .dockerignore

Without .dockerignore, COPY . . sends everything to the Docker daemon — including .git (potentially hundreds of MB), .env (secrets), node_modules, and test fixtures.

Mandatory exclusions:

.git
.gitignore
.env
.env.*
*.pyc
__pycache__
.venv
venv
node_modules
.npm
.idea
.vscode
*.swp
*.swo
tests/
test/
docs/
*.md
!README.md
docker-compose*.yml
Dockerfile*
.dockerignore
.coverage
htmlcov/
.pytest_cache/
.mypy_cache/
.ruff_cache/

Rules:

• Every project with a Dockerfile MUST have a .dockerignore
• Start restrictive, add exceptions with ! prefix as needed
• Test with: docker build --no-cache . and check context size in output
• Context size over 50MB usually means .dockerignore is missing entries

7. Health Checks

Without health checks, Docker (and orchestrators) have no way to know if your app is actually serving requests. A container can be "running" with a completely hung process.

HTTP health check (preferred for web services):

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
    CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]

For Node.js:

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
    CMD ["node", "-e", "fetch('http://localhost:3000/health').then(r => { if (!r.ok) process.exit(1) })"]

For distroless (no curl/wget):

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
    CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]

Tuning guidelines:

• --interval: 30s for most services, 10s for critical path
• --timeout: 5s usually sufficient; increase for services with slow health endpoints
• --start-period: Time for the app to boot. Set this generously (migrations, model loading)
• --retries: 3 is standard. Fewer for fast-failing services

Rules:

• Every production Dockerfile must have a HEALTHCHECK
• The health endpoint must verify real functionality (DB connection, not just "200 OK")
• Do not use curl in distroless images (it does not exist)
• Use the exec form CMD ["..."] not shell form CMD ...

8. Security Scanning

Every image must be scanned before deployment. Trivy is the industry standard — it is fast, free, and catches CVEs in OS packages and language dependencies.

Local scan:

trivy image --severity HIGH,CRITICAL your-image:tag

CI pipeline (fail the build on findings):

# GitHub Actions example
- name: Scan image with Trivy
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: ${{ env.IMAGE }}
    format: table
    exit-code: 1
    severity: HIGH,CRITICAL
    ignore-unfixed: true

Rules:

• Scan in CI on every build — not just occasionally
• Set --exit-code 1 so builds fail on HIGH/CRITICAL findings
• Use --ignore-unfixed to suppress CVEs with no available fix
• Pin the Trivy action/binary version for reproducible scans
• Scan both the builder stage and runtime stage if using multi-stage
• Maintain a .trivyignore file for accepted risks (with justification comments)

9. Compose Best Practices

Docker Compose for production requires explicit resource limits, health checks, and network isolation. The default "everything talks to everything" is a security problem.

Service dependencies with health checks:

services:
  api:
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy

Resource limits (prevent runaway containers):

services:
  api:
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: 512M
        reservations:
          cpus: "0.25"
          memory: 128M

Network isolation:

networks:
  frontend:
  backend:

services:
  api:
    networks: [frontend, backend]
  postgres:
    networks: [backend]  # Not accessible from frontend

Rules:

• Always set mem_limit / deploy resource limits — a memory leak should not take down the host
• Use restart: unless-stopped for production services
• Use named volumes for persistent data, never bind mounts in production
• Use env_file for environment variables, never hardcode secrets in compose files
• Pin image tags in compose, same as Dockerfiles
• Separate compose files for dev vs production: docker-compose.yml + docker-compose.prod.yml

10. Runtime Hardening

The image is only half the story. Runtime flags add defense-in-depth.

Read-only filesystem:

services:
  api:
    read_only: true
    tmpfs:
      - /tmp
      - /app/tmp

Drop capabilities:

services:
  api:
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE  # Only if binding to ports < 1024

No new privileges:

services:
  api:
    security_opt:
      - no-new-privileges:true

Docker run equivalent:

docker run \
  --read-only \
  --tmpfs /tmp \
  --cap-drop ALL \
  --security-opt no-new-privileges:true \
  --user 65532 \
  your-image:tag

Rules:

• Start with cap_drop: ALL, add back only what is needed
• no-new-privileges prevents setuid binaries from escalating
• Read-only filesystem forces you to explicitly declare writable paths
• Use tmpfs for directories that need to be writable (tmp, cache, pid files)

Anti-Patterns

These are the most common mistakes found in production Docker setups. Each one is a real incident waiting to happen.

Quick Reference

Build command (production):

DOCKER_BUILDKIT=1 docker build \
  --target runtime \
  --tag myapp:$(git rev-parse --short HEAD) \
  --label org.opencontainers.image.revision=$(git rev-parse HEAD) \
  --label org.opencontainers.image.created=$(date -u +"%Y-%m-%dT%H:%M:%SZ") \
  .

Verify non-root:

docker run --rm myapp:latest whoami
# Expected: nonroot or appuser, NEVER root

Check image size:

docker images myapp --format "{{.Repository}}:{{.Tag}} {{.Size}}"

Scan for secrets in layers:

docker history --no-trunc myapp:latest | grep -iE "(key|secret|password|token)"