Docker Networking

Docker Networking Troubleshooting

Diagnose and fix container networking issues: DNS resolution failures, outbound connectivity, inbound port access.

Trigger conditions

• Container can't resolve DNS (nslookup/dig timeout)
• Container can't reach external IPs (curl/wget timeout)
• Docker's embedded DNS (127.0.0.11) returns i/o timeout
• Let's Encrypt / ACME challenges failing from containers
• Containers can ping but not TCP/UDP

Quick Diagnostic Flow

Run these in order to isolate the problem:

# 1. Does the HOST have internet?
curl -sI --connect-timeout 5 http://1.1.1.1
nslookup google.com

# 2. Can a fresh alpine container reach the internet?
docker run --rm alpine wget -qO- --timeout=5 http://1.1.1.1
docker run --rm alpine nslookup google.com

# 3. Check what iptables backend Docker is using
readlink -f $(which iptables)
# /usr/sbin/xtables-nft-multi  → using iptables-nft (potential bug)
# /usr/sbin/xtables-legacy-multi → using iptables-legacy

# 4. Check if MASQUERADE is tracking packets
iptables -t nat -L POSTROUTING -n -v
# Look at the "pkts" counter for the bridge subnet rule
# If 0 packets despite container traffic, NAT is broken

# 5. Check if RELATED,ESTABLISHED is working
iptables -L DOCKER-CT -n -v
# Should show non-zero packet counts for ctstate RELATED,ESTABLISHED

Fix 1: iptables-legacy (MOST COMMON)

Symptom: Containers can ping external IPs but TCP/UDP (DNS, HTTP) all timeout. Host networking works fine.

Root cause: iptables-nft connection tracking doesn't properly track Docker bridge traffic. MASQUERADE happens but return packets aren't recognized as RELATED/ESTABLISHED.

Fix:

# Switch to iptables-legacy
update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

# Restart Docker to rebuild rules
systemctl restart docker

# Verify
docker run --rm alpine wget -qO- --timeout=5 http://1.1.1.1
docker run --rm alpine nslookup google.com

Persistence: The alternatives setting survives reboots but WILL be reset by iptables package updates (apt upgrade). Lock it permanently:

# Pin the package to prevent auto-switch on upgrade
apt-mark hold iptables

# Verify
readlink -f $(which iptables)
# Should show: /usr/sbin/xtables-legacy-multi

Note: ip6tables is bundled in the iptables package — pinning iptables covers both. To unpin later: apt-mark unhold iptables.

Affected setups: Common on Ubuntu 24.04+, Debian 12+, and VPS images where iptables defaults to nf_tables backend. Docker 26+ is particularly sensitive.

Fix 2: Hosting provider firewall

Symptom: Docker containers CAN reach the internet. Let's Encrypt validation fails with "Timeout during connect (likely firewall problem)". Localhost curl to port 80/443 works from the VPS, but external sources can't reach it.

Diagnosis:

# Test from the VPS itself (should work)
curl -sI http://localhost
curl -sI http://$(hostname -I | awk '{print $1}')

# Check UFW
ufw status
iptables -L ufw-user-input -n -v | grep -E 'dpt:80|dpt:443'

Root cause: Hosting provider (Hetzner, OVH, DigitalOcean, IP Connect, etc.) has a cloud firewall / security group that blocks inbound ports by default. Some budget providers (IP Connect, certain VPS resellers) block 80/443 to upsell their SSL certificate addon.

Fix A (preferred): Open ports 80 (HTTP) and 443 (HTTPS) in the provider's firewall panel. This is NOT a VM-level setting — it's in the provider's web console, often under "Firewall", "Security Groups", or "Network".

Fix B (workaround, no provider changes needed): Use DNS-01 ACME challenge instead of HTTP-01. This proves domain ownership via a DNS TXT record — Let's Encrypt never connects to your server. Requires a DNS provider with an API that Caddy supports (Cloudflare is free and well-supported). See references/dns01-workaround.md for full setup steps including Caddy DNS module configuration.

Fix C: Escalating to provider support (when they deny blocking)

Provider tier-1 support often checks the VPS's own UFW/iptables, sees ALLOW rules, and closes the ticket with "ports are open." This is wrong — the block is at their edge firewall, not your VPS. Use this diagnostic escalation to prove it:

Step 1: Multi-port scan from outside the VPS. Connect from your local machine (not through the VPS) and test multiple ports. If only SSH (22) works and everything else (80, 443, 25, 587, 8080) times out, it's provider-level filtering:

# From your LOCAL machine (not the VPS):
for port in 22 80 443 25 587 8080 8443; do
    timeout 5 bash -c "echo >/dev/tcp/YOUR_VPS_IP/$port" 2>&1 && echo "Port $port: OPEN" || echo "Port $port: BLOCKED"
done

Step 2: tcpdump on the VPS to prove packets never arrive. This is the definitive evidence — run a packet capture on the VPS's public interface while simultaneously trying to connect from outside:

# On the VPS (via SSH):
timeout 30 tcpdump -i eth0 -n -c 200 'tcp' 2>&1

While tcpdump runs, initiate connections to ports 80 and 443 from your local machine. If tcpdump shows ONLY port 22 traffic and zero packets on 80/443, the SYN packets are being dropped BEFORE they reach your server's network interface card. This rules out any VPS-side misconfiguration.

Step 3: Confirm VPS-side is clean. Document this for the support ticket:

# VPS firewall
ufw status
# VPS listeners
ss -tlnp | grep -E ':80 |:443 '
# Internal connectivity (proves the service works)
curl -sI http://localhost

Step 4: Send the evidence. The support message should include the tcpdump result (zero packets on the blocked ports), the multi-port scan (only 22 works), and confirmation that VPS UFW/listeners/internal-curl are all correct. The magic phrase is: "The packets are being dropped in the network layer above my VPS — please check your edge router or upstream firewall."

Note: Buying the provider's SSL certificate addon does NOT necessarily open the ports. Some budget providers sell SSL certs as a standalone product without touching their edge firewall rules. Confirm with support whether port opening is included or requires a separate request.

After escalating: set up a port-open watchdog (see references/port-watchdog.md) to monitor for when the provider actually unblocks the port. Provider support often fixes the issue hours later with no notification — the watchdog catches it within minutes at zero token cost.

Fix 4: Caddy ACME data clearing after IP/hostname change

Symptom: VPS IP address changed (provider reassignment, migration), ports are now open, but Caddy keeps retrying with old ACME staging data or targeting the old IP. Logs show "Timeout during connect (likely firewall problem)" referencing the OLD IP even though connectivity works on the new IP. Or Caddy uses acme-staging-v02 (staging endpoint) instead of production.

Root cause: Caddy caches ACME account data and cert locks on disk (/data/caddy/). After an IP change, stale staging accounts and locks persist in the Docker volume. Caddy may also still reference the old IP in its internal state.

Fix:

# 1. Stop Caddy
docker compose -f /path/to/deploy/docker-compose.yml down caddy

# 2. Remove stale staging ACME accounts and cert locks
#    (Docker volume paths vary — find with: docker volume inspect deploy_caddy-data)
rm -rf /var/lib/docker/volumes/deploy_caddy-data/_data/acme/acme-staging*
rm -f /var/lib/docker/volumes/deploy_caddy-data/_data/locks/issue_cert_*.lock

# 3. Bring Caddy back up — it will create a fresh production ACME account
docker compose -f /path/to/deploy/docker-compose.yml up -d caddy

# 4. Verify production endpoint (not staging):
docker compose logs caddy | grep 'acme-v02.api.letsencrypt.org'

After clearing: Caddy creates a new production ACME account and requests a fresh cert. The old production account (if one exists) in /data/caddy/acme/acme-v02.api.letsencrypt.org-directory/ is fine to keep — Caddy will use it. Only the staging data needs removal.

Pitfall: Do NOT restart the entire VPS — this is unnecessary. Only Caddy needs to be cycled. Restarting the VPS brings down the full Docker stack unnecessarily.

Fix 3: Docker daemon DNS config

Symptom: Container /etc/resolv.conf shows nameserver 127.0.0.11 with ExtServers configured, but DNS still fails. The dns: key in docker-compose.yml doesn't bypass the embedded DNS.

Root cause: Docker's dns: in compose adds forwarders to the embedded DNS (127.0.0.11), it doesn't write direct nameservers. If embedded DNS forwarding is broken, dns: in compose doesn't help.

Note: Fix #1 (iptables-legacy) usually resolves this since the embedded DNS just can't forward queries out. If DNS still fails after Fix #1, check /etc/docker/daemon.json:

{
  "dns": ["1.1.1.1", "8.8.8.8"]
}

Then systemctl restart docker.

Reference: SSH via Tor with password auth

When the VPS requires SSH access through Tor with password authentication from Hermes sessions (where /dev/tty isn't available):

# Create a password script
cat > /tmp/_ssh_pw.py << 'EOF'
#!/usr/bin/env python3
print("YOUR_PASSWORD")
EOF
chmod 700 /tmp/_ssh_pw.py

# SSH with ASKPASS
SSH_ASKPASS=/tmp/_ssh_pw.py \
SSH_ASKPASS_REQUIRE=force \
DISPLAY=:0 \
torsocks ssh -o StrictHostKeyChecking=no \
  -o UserKnownHostsFile=/dev/null \
  -o ConnectTimeout=20 \
  root@YOUR_IP "command"

# Clean up
rm /tmp/_ssh_pw.py

This works because SSH_ASKPASS with DISPLAY set forces SSH to use the password script instead of reading /dev/tty. The DISPLAY=:0 env var triggers this behavior even when there's no real X display.

For multiple commands in a session, use the bundled Python wrapper: scripts/torsocks-ssh.py. Import it with from torsocks_ssh import TorsocksSSH or run standalone with TORSOCKS_SSH_PW=pass python3 scripts/torsocks-ssh.py user@host "command".

Email Stack (Maddy)

For Maddy mail server TLS setup, DKIM key extraction, and DNS email records (SPF, DMARC, DKIM), see references/maddy-tls-setup.md. Covers self-signed cert generation, Maddy 0.9.5 config quirks (smtps/submission directives are silently ignored), and deliverability DNS records.

Pitfalls

• dns: in docker-compose doesn't bypass 127.0.0.11. It feeds external forwarders to Docker's embedded DNS. If embedded DNS is broken, dns: still fails.

• nc -u test can mislead. echo | nc -u -w3 1.1.1.1 53 returning "OK" only means the packet was SENT, not that a response was received. Use nslookup or dig for actual DNS tests.

• Ping working ≠ TCP/UDP working. ICMP can succeed while TCP/UDP fails, especially with broken conntrack. Test with curl/wget, not ping.

• Docker version matters. Docker 26-29 on iptables-nft backends has the most conntrack issues. Docker 24 on legacy iptables is generally fine.

• expose: in compose ≠ accessible from host. Docker Compose expose: only opens ports to linked containers, not to the host's localhost. Use docker inspect <container> --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' to find the container's bridge IP and connect directly. This is essential for tools like mc (MinIO client) that need to reach containers from the host.

• Minimal containers may lack basic tools. Alpine-based containers often don't have hostname, ping, dig, etc. Use docker inspect for IP discovery, not docker compose exec. Use wget (built into Alpine) instead of curl for HTTP tests from containers.

• Provider support checks the wrong firewall. When you report port blocking, tier-1 support often runs ufw status on your VPS, sees ALLOW, and says "ports are open." But the block is at their edge firewall (above your VPS), not your VPS's UFW. Use the escalation pattern in Fix 2C: multi-port scan + tcpdump on eth0 to prove packets never reach your server. Without this evidence, support will close the ticket as "no issue found."

• /etc/hosts workarounds break ACME. During initial setup, a common hack is adding 127.0.0.1 yourdomain.com to /etc/hosts so the VPS can reach itself via the domain name. After the real DNS record is fixed or the IP changes, this entry MUST be removed. Services use the VPS's own DNS resolution to determine which IP to expect ACME challenges from — a localhost override can cause challenge failures or cert mismatches. Check with cat /etc/hosts and remove any domain-to-localhost entries once real DNS is correct.

• Supabase self-hosted shim missing channel() causes white screen. When a Docker build swaps the real Supabase client for a self-hosted API shim, the shim must include a no-op channel() stub. Without it, any frontend code calling supabase.channel() (realtime subscriptions) throws TypeError: ke.channel is not a function and React renders a blank page. See references/supabase-shim-channel-stub.md for the stub code and full diagnosis.

• IP change requires DNS update + propagation time. When a provider reassigns your VPS IP, update the DNS A record immediately. Propagation to global resolvers takes minutes to hours (typically 5-30 min). Until propagation completes, external users may hit the old IP. Caddy's cert will still issue once Let's Encrypt's validators pick up the new DNS record — they use multiple geographically-distributed resolvers and often see the update faster than end users.