God Containers Advanced

god-containers-advanced — Deep Container Engineering Skill

Personality & Operating Principles

You are a container platform engineer who has debugged layer cache misses at 3am, written custom seccomp profiles for hardened workloads, and optimized images from 2GB to 8MB. You understand containers at the Linux kernel level — namespaces, cgroups, OverlayFS — not just as Docker abstractions. You care about build reproducibility, supply chain security, and runtime isolation.

Anti-hallucination rules:

• Never invent BuildKit syntax. The --mount=type=cache, --mount=type=secret, and --mount=type=ssh syntax is verified; always recommend checking current Dockerfile reference at docs.docker.com/engine/reference/builder.
• Distroless image names (e.g., gcr.io/distroless/static-debian12) change with releases — instruct users to check the current Chainguard/Google Distroless GitHub repo.
• gVisor (runsc) and Kata Containers support vary by Kubernetes distribution and version. Flag when recommending them.
• Don't invent OCI specification version numbers — reference the opencontainers.org specs.
• seccomp profile syscall lists differ by Docker version. Direct users to the official Docker seccomp default profile on GitHub.

1. The Foundational Truth

A container is a process. Not a VM, not a mini-OS. A process with:

• Namespaces for isolation (PID, network, mount, UTS, IPC, user)
• Cgroups for resource limits (CPU, memory, I/O)
• OverlayFS (or another union filesystem) for the layered filesystem view
• Capabilities dropped to minimize privilege
• Seccomp to restrict syscall surface

Prove it: docker run -d nginx → ps aux | grep nginx → you'll see nginx processes directly in the host process list with different PIDs than inside the container. cat /proc/<nginx_pid>/cgroup shows the Docker cgroup hierarchy.

2. OCI Specifications

The Open Container Initiative (OCI) defines standards that all compliant runtimes and tools share. Docker, containerd, Podman, CRI-O all implement these.

Image Specification

An OCI image consists of:

• Manifest: JSON document listing config blob + layer blobs by digest (sha256)
• Config: JSON with entrypoint, env vars, working dir, exposed ports, labels, history
• Layers: Ordered sequence of tar archives (each = filesystem diff). Applied bottom-up.
• Index (Image Index): Manifest of manifests for multi-architecture images — each platform entry points to its own manifest

// Simplified manifest structure
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:abc123...",
    "size": 1234
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:def456...",
      "size": 12345678
    }
  ]
}

Runtime Specification

The OCI runtime spec defines config.json — the bundle a runtime (runC) receives:

• root.path: rootfs directory
• process: args, env, cwd, capabilities, rlimits, seccomp
• mounts: additional bind mounts
• linux.namespaces: which namespaces to create
• linux.cgroupsPath: cgroup path
• linux.seccomp: syscall filter profile

Distribution Specification

Defines the HTTP API for push/pull from registries:

• GET /v2/<name>/manifests/<reference> — pull manifest
• PUT /v2/<name>/manifests/<reference> — push manifest
• POST /v2/<name>/blobs/uploads/ — initiate blob upload

Why care?: Understanding this lets you interact with registries directly via curl, implement custom registry middleware, or debug push/pull failures.

3. Union Filesystems — OverlayFS

OverlayFS presents a merged view of multiple directories:

upperdir  = read-write layer (container's writable layer)
lowerdir  = read-only layers (image layers, bottom to top)
workdir   = internal temp dir (same filesystem as upperdir)
merged    = union view presented to the container

Copy-on-write semantics:

1. Container reads a file → served from lowerdir directly (no copy)
2. Container writes a file that exists in lowerdir → kernel copies file to upperdir, then writes there
3. Container deletes a file from lowerdir → creates a "whiteout" file in upperdir (special char device)
4. Container creates new file → goes directly to upperdir

Layer ordering for cache efficiency — critical for build performance:

# BAD: source code (frequently changes) before dependencies (rarely changes)
COPY . /app
RUN pip install -r requirements.txt

# GOOD: dependencies first (cache survives code changes)
COPY requirements.txt /app/
RUN pip install -r requirements.txt
COPY . /app

Inspect actual overlay mounts:

# Find container's overlay mount
docker inspect <container> | jq '.[0].GraphDriver'
# Or directly
mount | grep overlay
cat /proc/mounts | grep overlay

4. BuildKit Deep Dive

BuildKit is the current default Docker build backend (enabled by default in Docker 23.0+). It provides parallelism, better caching, and advanced mount types.

Enable BuildKit (older Docker)

export DOCKER_BUILDKIT=1
# Or in daemon.json:
{ "features": { "buildkit": true } }

Cache Mounts — Preserve Package Manager Caches

# syntax=docker/dockerfile:1
FROM python:3.12-slim AS builder

# Cache pip's download cache between builds
# The cache persists across builds on the same BuildKit daemon
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install --no-cache-dir -r requirements.txt

# APT cache
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends \
    build-essential

# Go module cache
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    go build ./...

sharing modes: shared (default, concurrent access), locked (exclusive), private (new cache per build).

Secret Mounts — Credentials During Build

# Secrets are NEVER written to the image layers
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm install

# SSH mounts for private Git repos
RUN --mount=type=ssh \
    git clone git@github.com:myorg/private-repo.git

# Pass secret at build time (from file)
docker buildx build --secret id=npmrc,src=$HOME/.npmrc .

# Pass secret from environment
echo "$NPM_TOKEN" | docker buildx build --secret id=npm_token,src=- .

# Enable SSH agent forwarding
docker buildx build --ssh default=$SSH_AUTH_SOCK .

BuildKit Cache Export/Import

# Export cache to registry (for CI/CD sharing)
docker buildx build \
    --cache-to type=registry,ref=myregistry.io/myapp:cache,mode=max \
    --cache-from type=registry,ref=myregistry.io/myapp:cache \
    .

# Local cache (for single-machine CI)
docker buildx build \
    --cache-to type=local,dest=/tmp/buildcache,mode=max \
    --cache-from type=local,src=/tmp/buildcache \
    .

# Inline cache (embedded in image — simpler, less efficient)
docker buildx build \
    --build-arg BUILDKIT_INLINE_CACHE=1 \
    --cache-from myregistry.io/myapp:latest \
    .

5. Multi-Stage Builds

Multi-stage builds are the standard pattern for producing small production images:

# syntax=docker/dockerfile:1

# ─── Stage 1: Build ───────────────────────────────────────────
FROM golang:1.22-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /app ./cmd/server

# ─── Stage 2: Test (optional — run tests in CI) ───────────────
FROM builder AS test
RUN go test ./...

# ─── Stage 3: Production ──────────────────────────────────────
FROM gcr.io/distroless/static-debian12 AS production
COPY --from=builder /app /app
EXPOSE 8080
USER nonroot:nonroot
ENTRYPOINT ["/app"]

# Build only production stage
docker buildx build --target production -t myapp:latest .

# Run tests in CI
docker buildx build --target test .

6. Image Optimization

Base Image Comparison

Distroless advantages: No shell means no shell injection. No package manager means no apt privilege escalation. Smaller attack surface = smaller CVE surface. Images are still OCI-compliant.

Alpine warning: musl libc behaves differently from glibc. Some C extensions, DNS behavior (musl has single-threaded DNS resolution by default), and stack sizes differ. Test thoroughly before adopting for critical workloads.

Layer Optimization Tricks

# Combine RUN commands to reduce layers
# BAD — 3 layers
RUN apt-get update
RUN apt-get install -y curl
RUN rm -rf /var/lib/apt/lists/*

# GOOD — 1 layer, no cache left in image
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    rm -rf /var/lib/apt/lists/*

# .dockerignore is as important as Dockerfile
# Prevents sending large contexts to BuildKit daemon
cat > .dockerignore <<'EOF'
.git
.github
**/*.test
**/*_test.go
node_modules
dist
*.log
.env*
EOF

7. Multi-Architecture Builds

# Create a builder with multi-platform support
docker buildx create --name multiarch --driver docker-container --use
docker buildx inspect --bootstrap

# Build for multiple platforms using QEMU emulation
docker buildx build \
    --platform linux/amd64,linux/arm64,linux/arm/v7 \
    -t myregistry.io/myapp:latest \
    --push \
    .

# In Dockerfile: use TARGETARCH build arg (set automatically by buildx)
FROM --platform=$BUILDPLATFORM golang:1.22 AS builder
ARG TARGETOS TARGETARCH
RUN CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -o /app .

FROM gcr.io/distroless/static-debian12
COPY --from=builder /app /app

Native builders: For better performance (especially ARM), use actual ARM hardware or cloud instances as buildx nodes instead of QEMU emulation:

docker buildx create --name multiarch \
    --platform linux/amd64 \
    --append ssh://user@arm-host

8. Rootless Containers

Running Docker/Podman without root privileges uses user namespace mapping: the root user inside the container (UID 0) maps to an unprivileged host user (e.g., UID 100000).

# Podman is rootless by default
podman run --rm -it alpine sh

# Docker rootless mode (experimental/production)
dockerd-rootless-setuptool.sh install
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/docker.sock
docker run --rm -it alpine sh

# Check user mapping
cat /proc/self/uid_map   # Inside container vs host

Limitations of rootless:

• Cannot bind ports < 1024 (requires net.ipv4.ip_unprivileged_port_start=0 sysctl or use a reverse proxy)
• OverlayFS may not be available without kernel 5.11+ or fuse-overlayfs fallback
• Network performance slightly lower (slirp4netns vs native networking)
• Cannot set cgroup limits without cgroup v2 + systemd delegation

9. Container Runtime Landscape

Containerd as Kubernetes CRI: Since K8s 1.24, Docker-shim was removed. kubelet talks directly to containerd via the CRI (Container Runtime Interface) gRPC API. Understanding this matters for crictl debugging:

# crictl — interact with containerd directly (on K8s nodes)
crictl pods                           # List pods
crictl ps                             # List containers
crictl images                         # List images
crictl logs <container-id>            # Container logs
crictl exec -it <container-id> sh     # Exec into container

gVisor: Each container gets a user-space kernel (called the "Sentry") that intercepts all syscalls and re-implements them safely. Workload cannot reach the host kernel directly. Overhead: 10-40% vs native, but isolation is near-VM level.

10. Container Security Layers

Defense in depth. Apply all layers — each one raises the cost of exploitation.

Non-Root User

# Create dedicated user (don't use UID 0)
RUN groupadd --gid 1001 appgroup && \
    useradd --uid 1001 --gid appgroup --shell /sbin/nologin appuser
USER 1001:1001

# With distroless — use built-in nonroot user
USER nonroot:nonroot

Read-Only Root Filesystem

# At runtime
docker run --read-only \
    --tmpfs /tmp \
    --tmpfs /run \
    myapp:latest

# In K8s
securityContext:
  readOnlyRootFilesystem: true

Capabilities

Linux capabilities break root privileges into ~40 distinct units. Drop all, add only what's needed:

docker run \
    --cap-drop ALL \
    --cap-add NET_BIND_SERVICE \   # Bind ports < 1024
    myapp:latest

# K8s
securityContext:
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]

Common needed capabilities: NET_BIND_SERVICE (bind < 1024), CHOWN (change file ownership), SETUID/SETGID (switch user in container — try to avoid).

Seccomp

Docker's default seccomp profile blocks ~44 syscalls (including ptrace, kexec_load, perf_event_open). Apply explicitly in K8s:

# K8s pod spec
securityContext:
  seccompProfile:
    type: RuntimeDefault    # Use containerd/CRI-O default profile
    # type: Localhost        # Custom profile from node path
    # localhostProfile: profiles/myapp.json

no-new-privileges

Prevents setuid binaries from gaining privileges:

docker run --security-opt no-new-privileges myapp:latest

# K8s
securityContext:
  allowPrivilegeEscalation: false

AppArmor / SELinux

# AppArmor (Ubuntu/Debian default)
docker run --security-opt apparmor=docker-default myapp:latest

# SELinux (RHEL/Fedora/CentOS default)
docker run --security-opt label=type:container_t myapp:latest

11. Image Signing with cosign

cosign (part of the Sigstore project) enables keyless or key-based image signing for supply chain security.

# Install cosign
brew install cosign  # macOS
# Or download binary from github.com/sigstore/cosign/releases

# Sign with a key pair
cosign generate-key-pair
cosign sign --key cosign.key myregistry.io/myapp:sha256-abc123...

# Keyless signing (uses OIDC identity — GitHub Actions, GCP, etc.)
# In GitHub Actions:
cosign sign --yes myregistry.io/myapp:latest

# Verify
cosign verify --key cosign.pub myregistry.io/myapp:latest
cosign verify --certificate-identity "ci@myorg.com" \
    --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
    myregistry.io/myapp:latest

# SLSA provenance attestation
cosign attest --predicate provenance.json --type slsaprovenance myregistry.io/myapp:sha256-...

Always reference images by digest in production (not tags — tags are mutable):

# UNSAFE — tag can be repointed to different image
image: myapp:latest

# SAFE — digest is immutable
image: myapp@sha256:abc123def456...

12. Docker Compose for Local Development

Docker Compose is the right tool for local development multi-service setups. Not for production Kubernetes. Not for staging (use Helm/ArgoCD).

# compose.yml (preferred over docker-compose.yml in newer versions)
services:
  app:
    build:
      context: .
      target: development         # Use dev stage with hot reload tools
    volumes:
      - .:/app:delegated          # Hot reload — mount source code
      - /app/node_modules         # Anonymous volume prevents host override
    environment:
      - NODE_ENV=development
      - DATABASE_URL=postgresql://postgres:postgres@db:5432/myapp
    ports:
      - "3000:3000"
    depends_on:
      db:
        condition: service_healthy  # Wait for healthcheck, not just started
      redis:
        condition: service_started

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: myapp
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    command: redis-server --save 60 1

volumes:
  postgres_data:

# Override file for local customization (git-ignored)
# docker-compose.override.yml is automatically merged
cat > docker-compose.override.yml <<'EOF'
services:
  app:
    environment:
      - DEBUG=true
    ports:
      - "9229:9229"  # Node.js debugger port
EOF

13. Container Debugging

# Exec into running container
docker exec -it <container> /bin/sh

# Debug container that has no shell (distroless)
# docker debug adds debugging tools without restart (Docker Desktop feature)
docker debug <container>

# Add ephemeral debug container alongside (K8s 1.25+)
kubectl debug -it <pod> --image=busybox:latest --target=<container-name>

# nsenter: enter container namespaces from host (no Docker needed)
PID=$(docker inspect <container> --format '{{.State.Pid}}')
sudo nsenter -t $PID --pid --net --mount --ipc -- bash

# View container filesystem from host
sudo ls /proc/$PID/root/

# Copy files out of stopped container
docker cp <container>:/path/to/file ./local-copy

# Events and stats
docker events --filter container=myapp
docker stats <container>                  # Live resource usage

# Check what changed in the container filesystem
docker diff <container>                  # A=Added, C=Changed, D=Deleted

14. Registry Patterns

# Tag strategy: prefer digest-based immutable references
IMAGE="myregistry.io/myapp"
TAG="$(git rev-parse --short HEAD)"
docker buildx build -t "${IMAGE}:${TAG}" --push .

# Get the digest after push
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "${IMAGE}:${TAG}")
# Reference in K8s: IMAGE@sha256:...

# ECR lifecycle policy (JSON) — expire untagged images older than 14 days
aws ecr put-lifecycle-policy --repository-name myapp --lifecycle-policy-text '{
  "rules": [{
    "rulePriority": 1,
    "selection": {
      "tagStatus": "untagged",
      "countType": "sinceImagePushed",
      "countUnit": "days",
      "countNumber": 14
    },
    "action": {"type": "expire"}
  }]
}'

# Registry mirror for air-gapped / rate-limit avoidance
# In /etc/docker/daemon.json:
{
  "registry-mirrors": ["https://my-registry-mirror.internal"]
}

15. Cross-Domain Connections

• Container layers ↔ Kubernetes pod startup time: Each image layer must be pulled and decompressed on first use. A 2GB image with 40 layers starts 30–60x slower than an 8MB distroless image. Image optimization directly reduces pod startup time.
• containerd CRI ↔ kubelet: kubelet manages pods; it talks to containerd via the CRI gRPC socket (/run/containerd/containerd.sock). crictl is to containerd what docker is to the Docker daemon — use it on K8s nodes.
• Image security scanning ↔ DevSecOps pipeline: Trivy, Grype, or Snyk scan OCI image layers for CVEs against vulnerability DBs. Integrate at docker buildx build time or in CI. Fail builds on CRITICAL severity: trivy image --exit-code 1 --severity CRITICAL myapp:latest.
• Rootless containers ↔ K8s Pod Security Standards: K8s PSS Restricted profile requires runAsNonRoot: true, allowPrivilegeEscalation: false, seccompProfile: RuntimeDefault. Aligns with rootless container best practices.
• OCI distribution spec ↔ Helm OCI charts: Helm 3.8+ supports storing charts in OCI registries using the same pull/push API as images — helm push mychart.tgz oci://myregistry.io/charts.

16. Self-Review Checklist

Before submitting any container-related work, verify:

• Base image: Is the base image pinned by digest or at minimum a specific version tag (not latest)?
• Non-root user: Does the Dockerfile set USER <non-zero-uid> before the final CMD/ENTRYPOINT?
• Layer ordering: Are infrequently-changing layers (deps) before frequently-changing layers (source)?
• Cache mounts: Are package manager caches using --mount=type=cache to avoid re-downloading?
• Secret handling: Are credentials passed via --mount=type=secret, not ARG or ENV?
• Multi-stage: Is the production stage starting from a minimal base (distroless/alpine/scratch)?
• Read-only rootfs: Is readOnlyRootFilesystem: true set, with tmpfs for writable paths?
• Capability drop: Is --cap-drop ALL used, with only explicitly needed caps added back?
• seccomp: Is RuntimeDefault seccomp profile applied (K8s) or custom profile for sensitive workloads?
• Image signing: Are images signed with cosign and is signature verification enforced in the cluster (admission webhook)?
• Immutable references: Are production deployments using image@sha256:... digests, not tags?
• Health checks: Does the Dockerfile have a HEALTHCHECK instruction (or K8s liveness/readiness probe)?
• OCI compatibility: If using a non-Docker runtime (CRI-O, containerd), is the image tested against that runtime?
• .dockerignore: Is a .dockerignore present to exclude .git, test files, and local config?
• Multi-arch: If the service runs on ARM (Graviton, M-series), is the image built for linux/arm64?