Caddy Https Troubleshoot

Troubleshoot HTTPS Skill

Systematic diagnosis and resolution of HTTPS/SSL certificate issues in the Caddy reverse proxy with Cloudflare DNS-01 challenge.

Quick Start

Run the diagnostic script to identify issues:

/home/dawiddutoit/projects/network/.claude/skills/troubleshoot-https/scripts/diagnose-https.sh

Or follow the step-by-step diagnosis in Instructions.

Table of Contents

1. When to Use This Skill
2. What This Skill Does
3. Instructions
   • 3.1 Check API Key Configuration
   • 3.2 Verify Cloudflare DNS Plugin
   • 3.3 Analyze Caddy Logs
   • 3.4 Validate Caddyfile Syntax
   • 3.5 Test Certificate Validity
   • 3.6 Diagnose Specific Error
   • 3.7 Apply Fix
4. Supporting Files
5. Expected Outcomes
6. Common Error Reference
7. Requirements
8. Red Flags to Avoid

1. When to Use This Skill

Explicit Triggers:

• "HTTPS not working"
• "SSL certificate error"
• "Certificate not obtained"
• "Caddy keeps restarting"
• "Fix HTTPS certificates"
• "Invalid format for Authorization header"

Implicit Triggers:

• Browser shows "Your connection is not private"
• Services accessible via HTTP but not HTTPS
• Caddy container in restart loop
• New domain not getting certificate

Debugging Triggers:

• "Why is my certificate expired?"
• "Why can't Caddy get a certificate?"
• "Cloudflare DNS challenge failing"

2. What This Skill Does

1. Checks API Key - Verifies CLOUDFLARE_API_KEY is set and passed to container
2. Verifies Plugin - Confirms Cloudflare DNS plugin is compiled into Caddy
3. Analyzes Logs - Searches for certificate errors in Caddy logs
4. Validates Config - Tests Caddyfile syntax
5. Tests Certificates - Checks certificate validity for each domain
6. Identifies Error - Matches symptoms to known issues
7. Provides Fix - Gives specific commands to resolve the issue

3. Instructions

3.1 Check API Key Configuration

Step 1: Verify API key is set in .env

grep "CLOUDFLARE_API_KEY" /home/dawiddutoit/projects/network/.env | head -1

Expected: CLOUDFLARE_API_KEY="Xv5MOdOT... (starts with alphanumeric, not v4:)

Step 2: Verify API key is passed to container

docker exec caddy env | grep CLOUDFLARE_API_KEY

Expected: Shows the API key value (not empty)

If empty or missing:

# Recreate container to pick up .env changes
docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml up -d --force-recreate caddy

Key distinction:

• API Token format: Xv5MOdOT... (alphanumeric string) - CORRECT
• Global API Key format: v4:... or long hex string - WRONG

3.2 Verify Cloudflare DNS Plugin

docker exec caddy caddy list-modules | grep cloudflare

Expected: dns.providers.cloudflare

If not present: Plugin not compiled into Caddy. Rebuild:

cd /home/dawiddutoit/projects/network && \
docker compose build --no-cache caddy && \
docker compose up -d --force-recreate caddy

3.3 Analyze Caddy Logs

Check for certificate-related messages:

docker logs caddy 2>&1 | grep -i -E "certificate|error|failed|invalid" | tail -30

Look for success messages:

docker logs caddy 2>&1 | grep "certificate obtained successfully"

Expected for each domain: certificate obtained successfully {"identifier": "domain.temet.ai"}

3.4 Validate Caddyfile Syntax

docker exec caddy caddy validate --config /etc/caddy/Caddyfile

Expected: Valid configuration (no errors)

If syntax error:

1. Read the Caddyfile to identify the error
2. Fix the syntax issue
3. Reload Caddy

3.5 Test Certificate Validity

Test a single domain:

echo | openssl s_client -servername pihole.temet.ai -connect pihole.temet.ai:443 2>/dev/null | openssl x509 -noout -dates -issuer

Expected output:

notBefore=<date>
notAfter=<date>
issuer=C = US, O = Let's Encrypt, CN = R3

Test all domains:

for domain in pihole jaeger langfuse sprinkler ha; do
  echo "=== $domain.temet.ai ==="
  echo | openssl s_client -servername $domain.temet.ai -connect $domain.temet.ai:443 2>/dev/null | \
    openssl x509 -noout -dates 2>&1 || echo "FAILED to get certificate"
  echo
done

3.6 Diagnose Specific Error

Match error message to diagnosis:

3.7 Apply Fix

Fix A: Wrong API Key Type

The error "Invalid format for Authorization header" means you're using the Global API Key instead of an API Token.

1. Create new API Token:

   • Go to: https://dash.cloudflare.com/profile/api-tokens
   • Click "Create Token"
   • Select template: "Edit zone DNS"
   • Zone Resources: Include -> Specific zone -> temet.ai
   • Click "Continue to summary" -> "Create Token"

2. Update .env:

   # Edit .env and replace CLOUDFLARE_API_KEY with the new token
   nano /home/dawiddutoit/projects/network/.env
   

3. Recreate Caddy:

   docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml up -d --force-recreate caddy
   

4. Verify:

   docker logs caddy 2>&1 | grep "certificate obtained successfully"
   

Fix B: Environment Variable Not Set

1. Verify .env has the key:

   grep CLOUDFLARE_API_KEY /home/dawiddutoit/projects/network/.env
   

2. Verify docker-compose.yml passes it:

   grep -A5 "caddy:" /home/dawiddutoit/projects/network/docker-compose.yml | grep CLOUDFLARE
   

3. Recreate container:

   docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml up -d --force-recreate caddy
   

Fix C: Cloudflare Plugin Not Compiled

cd /home/dawiddutoit/projects/network && \
docker compose build --no-cache caddy && \
docker compose up -d --force-recreate caddy

Build takes approximately 5 minutes on Raspberry Pi.

Fix D: Rate Limit or DNS Propagation

1. Wait 5 minutes for DNS propagation

2. Restart Caddy:

   docker compose -f /home/dawiddutoit/projects/network/docker-compose.yml restart caddy
   

3. If still failing, check Let's Encrypt rate limits:

   • Limit: 50 certificates per domain per week
   • Check: https://crt.sh/?q=temet.ai

Fix E: API Token Missing Permissions

1. Go to: https://dash.cloudflare.com/profile/api-tokens
2. Find your token and click "Edit"
3. Verify permissions:
   • Zone -> DNS -> Edit (required)
   • Zone Resources -> Include -> Specific zone -> temet.ai
4. If missing, create new token with correct permissions

Fix F: Caddyfile Syntax Error

1. Check error in logs:

   docker logs caddy 2>&1 | tail -50
   

2. Validate Caddyfile:

   docker exec caddy caddy validate --config /etc/caddy/Caddyfile
   

3. Common syntax issues:

   • Missing closing braces }
   • Invalid directive names
   • Wrong environment variable syntax

4. Fix the Caddyfile and reload:

   docker exec caddy caddy reload --config /etc/caddy/Caddyfile
   

4. Supporting Files

5. Expected Outcomes

Success:

• All diagnostic checks pass
• certificate obtained successfully for each domain
• HTTPS working with valid Let's Encrypt certificates
• Certificate shows issuer: C = US, O = Let's Encrypt

Partial Success:

• Issue identified but fix requires manual action (e.g., create new API token)
• Certificate pending (DNS propagation in progress)

Failure Indicators:

• API key format is wrong (Global API Key vs API Token)
• Cloudflare plugin not compiled into Caddy
• Caddyfile syntax errors blocking startup
• Let's Encrypt rate limit exceeded

6. Common Error Reference

7. Requirements

• Docker running with Caddy container
• Valid Cloudflare API Token with "Edit zone DNS" permission
• .env file with CLOUDFLARE_API_KEY set
• Internet access for Cloudflare API calls

8. Red Flags to Avoid

• Do not use Global API Key (format: v4:... or long hex) - use API Token instead
• Do not skip verifying the plugin is compiled before troubleshooting further
• Do not delete caddy_data volume unless absolutely necessary (certificates stored there)
• Do not exceed Let's Encrypt rate limits (50 certs/domain/week)
• Do not forget to recreate container after .env changes
• Do not use --no-verify to bypass certificate errors
• Do not commit API tokens to git

Notes

• Certificates are stored in Docker volume network_caddy_data
• Certificates auto-renew 30 days before expiry
• DNS-01 challenge allows certificates for internal-only services
• Caddy checks renewals every 12 hours
• Build with --no-cache if plugin changes aren't taking effect
• API Token (not Global API Key) is required for Caddy Cloudflare DNS plugin