Docker Errors Runtime

docker-errors-runtime

Quick Reference

Exit Code Reference

Critical Warnings

NEVER ignore exit code 137 — it ALWAYS indicates the container was forcefully killed. Check OOM events with docker inspect and dmesg before increasing memory limits blindly.

NEVER use --oom-kill-disable without setting a memory limit (-m) — the container can consume ALL host memory and crash the entire system.

NEVER assume a container that exits with code 0 is healthy — it may have completed a one-shot command instead of running as a long-lived service. ALWAYS verify the process runs in the foreground.

ALWAYS check docker logs before any other debugging step — 90% of runtime issues are explained in the application output.

ALWAYS use docker inspect --format='{{.State.ExitCode}}' to get the exact exit code — docker ps -a truncates status information.

Debugging Workflow

Step 1: Check Logs

# Last 100 lines
docker logs --tail 100 <container>

# Follow live output with timestamps
docker logs -f -t <container>

# Logs from last 5 minutes
docker logs --since 5m <container>

Step 2: Inspect Container State

# Exit code and error message
docker inspect --format='{{.State.ExitCode}}' <container>
docker inspect --format='{{.State.Error}}' <container>

# OOM killed?
docker inspect --format='{{.State.OOMKilled}}' <container>

# Full state as JSON
docker inspect --format='{{json .State}}' <container> | jq .

Step 3: Exec Into Running Container

# Interactive shell (if container is still running)
docker exec -it <container> sh
docker exec -it <container> bash

# Check filesystem, processes, network
docker exec <container> ps aux
docker exec <container> df -h
docker exec <container> cat /etc/resolv.conf

Step 4: Check System Events

# Events for specific container in last 10 minutes
docker events --since 10m --filter container=<container>

# OOM events specifically
docker events --filter event=oom --since 1h

# All die events
docker events --filter event=die --since 1h

Step 5: Resource Usage

# Live resource stats
docker stats <container>

# Single snapshot
docker stats --no-stream <container>

# System-wide disk usage
docker system df -v

Runtime Error Diagnostic Table

Container Exits Immediately (Exit Code 0 or 1)

OOM Killed (Exit Code 137)

Permission Denied

Port Already in Use

Exec Format Error

Read-Only Filesystem

PID Limit and Resource Exhaustion

docker inspect for Debugging

Essential Inspect Commands

# Full state overview
docker inspect --format='{{json .State}}' <container> | jq .

# Why did it stop?
docker inspect --format='ExitCode={{.State.ExitCode}} OOM={{.State.OOMKilled}} Error={{.State.Error}}' <container>

# What command is it running?
docker inspect --format='Entrypoint={{.Config.Entrypoint}} Cmd={{.Config.Cmd}}' <container>

# Environment variables
docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' <container>

# Mount points
docker inspect --format='{{range .Mounts}}{{.Type}}: {{.Source}} -> {{.Destination}} ({{if .RW}}rw{{else}}ro{{end}}){{println}}{{end}}' <container>

# Network settings
docker inspect --format='{{range $net, $config := .NetworkSettings.Networks}}{{$net}}: {{$config.IPAddress}}{{println}}{{end}}' <container>

# Resource limits
docker inspect --format='Memory={{.HostConfig.Memory}} CPUs={{.HostConfig.NanoCpus}} PidsLimit={{.HostConfig.PidsLimit}}' <container>

# Health check status
docker inspect --format='{{.State.Health.Status}}' <container>
docker inspect --format='{{json .State.Health}}' <container> | jq .

# Restart count
docker inspect --format='RestartCount={{.RestartCount}}' <container>

Decision Trees

Container Won't Start

Container won't start
├─ Exit 125 → Docker daemon error
│  ├─ "invalid reference format" → Fix image name/tag
│  ├─ "no such image" → Pull image first: docker pull <image>
│  └─ "invalid mount config" → Fix volume/mount syntax
├─ Exit 126 → Command not executable
│  ├─ Check file permissions → chmod +x
│  └─ Check binary format → file <binary>
├─ Exit 127 → Command not found
│  ├─ Typo in CMD/ENTRYPOINT → Fix spelling
│  ├─ Binary not in PATH → Use absolute path
│  └─ Binary not installed → Add to Dockerfile
└─ Exit 0/1 instantly → See "Container Exits Immediately" table

Container Crashes After Running

Container was running, then died
├─ Exit 137 → Killed
│  ├─ OOMKilled=true → Memory limit too low (see OOM section)
│  ├─ OOMKilled=false, after docker stop → Stop timeout too short
│  └─ OOMKilled=false, unexpected → Check docker events + dmesg
├─ Exit 139 → Segfault
│  ├─ Native library issue → Check library compatibility
│  └─ Memory corruption → Debug with --cap-add SYS_PTRACE
├─ Exit 143 → Graceful SIGTERM
│  └─ Expected from docker stop → Normal shutdown
└─ Exit 1 → Application error
   └─ Check docker logs → Fix application bug

Reference Links

• references/diagnostics.md -- Complete error-to-solution mapping for all runtime errors
• references/examples.md -- Step-by-step debugging sessions with real commands
• references/anti-patterns.md -- Runtime configuration mistakes and how to avoid them

Official Sources

• https://docs.docker.com/engine/daemon/troubleshoot/
• https://docs.docker.com/reference/cli/docker/container/run/
• https://docs.docker.com/reference/cli/docker/container/logs/
• https://docs.docker.com/reference/cli/docker/inspect/
• https://docs.docker.com/reference/cli/docker/system/events/