Docker Build and Test Workflow

Docker Build and Test Workflow

This skill provides guidance for efficiently building, testing, and working with Docker images in this repository.

Core Principles

1. Scripts rebuild by design - The testing scripts prioritize reliability over speed by rebuilding images
2. Docker layer caching is your friend - Rebuilds are fast when only small changes occur
3. Avoid unnecessary script invocations - Reuse existing images when possible
4. Manual cache management - Only delete images as a last resort

Understanding Script Behavior

What the Scripts Actually Do

scripts/build-local.sh [target]

• Builds specified target (standard, dev, or stage)
• Relies on Docker layer caching for speed
• Securely handles GitHub token via gh auth
• Always performs a build, but uses cache when available

scripts/test-dockerfile.sh [target|dockerfile] [--cleanup]

• For base targets (standard/dev): Always rebuilds via build-local.sh
• For cookbooks: Smart about base (ensure_base_image() checks timestamps), but always rebuilds cookbook
• Runs comprehensive goss validation tests
• Keeps test images by default (use --cleanup to remove)

scripts/shell.sh [target|cookbook] [tag]

• Always rebuilds the specified target or cookbook
• Launches interactive shell in the container
• Auto-cleans container on exit (but keeps image)

The One Smart Cache Check

The ensure_base_image() function (used for cookbook testing):

• Checks if agentic-container:latest exists
• Compares Dockerfile modification time vs image creation time
• Only rebuilds base if Dockerfile is newer or image missing
• This is the ONLY automatic cache optimization in the scripts

Efficient Workflow Patterns

Pattern 1: Initial Build + Iterative Testing

# Step 1: Initial build and test (will rebuild)
./scripts/test-dockerfile.sh standard

# Step 2: Iterate on goss tests WITHOUT rebuilding
# Edit goss/standard.yaml, then:
docker run --rm --user root \
  -v "$PWD/goss/base-common.yaml:/tmp/goss-base-common.yaml:ro" \
  -v "$PWD/goss/standard.yaml:/tmp/goss-base.yaml:ro" \
  test-standard:latest \
  bash -c 'goss -g /tmp/goss-base-common.yaml -g /tmp/goss-base.yaml validate --format documentation'

# Step 3: Final test with script (rebuilds to verify)
./scripts/test-dockerfile.sh standard --cleanup

Pattern 2: Check Before Rebuilding

# Check what images exist
docker images | grep -E '(agentic|test-)'

# If test-standard:latest exists, use it directly
docker run --rm test-standard:latest python --version

# Only rebuild if you changed the Dockerfile
./scripts/build-local.sh standard agentic-container:latest

Pattern 3: Cookbook Development

# Initial build and test (smart about base, rebuilds cookbook)
./scripts/test-dockerfile.sh docs/cookbooks/python-cli/Dockerfile

# The script built: test-dockerfile-TIMESTAMP image
# Find it:
docker images | grep test-dockerfile

# Debug with existing image (no rebuild)
docker run --rm -it test-dockerfile-1234567890 bash

# When done iterating, run final test with cleanup
./scripts/test-dockerfile.sh docs/cookbooks/python-cli/Dockerfile --cleanup

Pattern 4: Quick Validation

# Instead of shell.sh (which rebuilds), use existing image:
docker run --rm -it test-standard:latest bash

# Or run a quick command:
docker run --rm test-standard:latest mise list

Pattern 5: Prototyping Dockerfile Commands

When adding complex commands to Dockerfile, test them first in existing image:

# Step 1: Test interactively to develop the command
docker run --rm -it --user root test-dev:latest bash
# Inside container, try commands until they work:
# $ mkdir -p /some/path
# $ ln -sf source target
# $ command --version
# $ exit

# Step 2: Test as one-liner (how Dockerfile RUN works)
docker run --rm --user root test-dev:latest bash -c '
mkdir -p /some/path && \
ln -sf source target && \
command --version
'

# Step 3: If successful, add to Dockerfile
RUN mkdir -p /some/path \
    && ln -sf source target

# Step 4: Rebuild and verify changes persisted
./scripts/test-dockerfile.sh dev
docker run --rm test-dev:latest command --version

Why This Matters:

• Catches path issues, permission problems, and syntax errors before build
• Validates that commands work in the actual environment
• Much faster than rebuild cycles for iteration
• Helps understand what files/directories already exist

Common Use Cases:

• Creating symlinks (test paths are correct)
• Setting up configuration files
• Verifying package installations work
• Testing permission requirements

Pattern 6: Debugging Multi-Stage Builds

When packages or files are missing from final image but work in build stage:

# Step 1: Build the intermediate stage directly
./scripts/build-local.sh npm-globals-stage test-npm-globals:latest

# Step 2: Inspect what the stage actually contains
docker run --rm test-npm-globals:latest bash -c 'ls -la /path/to/expected/files'

# Step 3: Check for symlinks (Docker COPY follows them!)
docker run --rm test-npm-globals:latest bash -c 'ls -la /path/to/bin/'

# Step 4: Compare to final image
docker run --rm test-dev:latest bash -c 'ls -la /path/to/bin/'

# Step 5: Identify what's different
# - Symlinks become regular files when COPY'd
# - Permissions may change
# - Files might be in different locations

Common Multi-Stage Issues:

• Docker COPY follows symlinks - Converts them to regular files, breaking relative paths
• Files exist in stage but not in final - Check COPY commands copy the right paths
• Permissions change between stages - RUN commands in final stage may run as different user
• ARG not passed to stage - Re-declare ARG in each stage that needs it

Solution Patterns:

• For symlinks: Recreate them in final stage with RUN command
• For missing files: Verify COPY source path includes the files
• For permissions: Run chmod in final stage after COPY

When Rebuilds Are Required

You MUST Rebuild When:

• Dockerfile content changed (RUN, COPY, ADD, etc.)
• ARG versions updated (NODE_VERSION, PYTHON_VERSION, etc.)
• Base image dependencies changed
• Files copied into image (COPY/ADD) have changed

Layer Cache Makes These Fast When:

• Only bottom layers changed (Dockerfile ordered well)
• System packages unchanged (apt-get install cached)
• Language installations unchanged (mise installations cached)
• Only scripts or configs changed

You DON'T Need to Rebuild When:

• Only goss test files changed (mount them at test time)
• Documentation changed (*.md files)
• CI configuration changed (.github/workflows/*.yml)
• Comments in Dockerfile changed

Layer Cache Optimization

Understanding Image vs Layer Cache

CRITICAL: Deleting an image DOES NOT clear layer cache!

# This removes the image tag but layers persist:
docker rmi test-dev:latest

# Build will still show CACHED for unchanged layers:
./scripts/test-dockerfile.sh dev

# To actually clear layer cache:
docker builder prune -f

# Nuclear option (clears everything including unused images):
docker system prune -f && docker builder prune -f

Why This Matters:

• Docker stores layers separately from image tags
• Multiple images can share the same layers
• Removing an image only removes the tag, not the layers
• Layer cache persists across branches and image deletions

Understanding Layer Invalidation

Docker rebuilds from the first changed layer onward. Order matters:

# ✅ GOOD: Infrequently changing items first
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y curl
ARG NODE_VERSION=24.8.0
RUN mise use -g node@${NODE_VERSION}
COPY scripts/ /usr/local/bin/  # Changes frequently

# ❌ BAD: Frequently changing items first
FROM ubuntu:24.04
COPY scripts/ /usr/local/bin/  # Changes frequently, invalidates all below
RUN apt-get update && apt-get install -y curl

Checking Cache Effectiveness

# Watch build output for "CACHED" vs "RUN" steps
./scripts/build-local.sh standard agentic-container:test | grep -E '(CACHED|RUN|COPY)'

# If you see mostly CACHED, layer cache is working well
# If you see mostly RUN, something early in Dockerfile changed

When Layer Cache Causes Problems

Symptom: Build shows "CACHED" but you changed the Dockerfile Cause: Layer hash collision or cache from different branch/state

Solutions:

1. Make a small change to force cache bust - Add/modify a comment in the RUN command
2. Clear builder cache - docker builder prune -f (fast, selective)
3. Use --no-cache - Only as last resort (slowest, rebuilds everything)

Example Cache Bust:

# Before (keeps showing CACHED even after adding commands):
RUN mise use -g node@${NODE_VERSION} \
    && your-new-commands-here

# After (change comment to bust cache):
# v2: Added symlink creation
RUN mise use -g node@${NODE_VERSION} \
    && your-new-commands-here

Verifying Changes Persisted:

# Check if your changes made it into the image:
docker history test-dev:latest --no-trunc | grep "your-command"

# Or inspect the actual files:
docker run --rm test-dev:latest ls -la /path/to/your/files

Working with Test Images

Test Image Naming Conventions

• Base targets in local mode: test-standard:latest, test-dev:latest
• Cookbooks in local mode: test-dockerfile-TIMESTAMP (timestamped)
• CI mode: Pre-built images with specific tags

Retaining vs Cleaning Up

# Default: Keep test images for inspection
./scripts/test-dockerfile.sh standard
docker run --rm -it test-standard:latest bash  # Debug it

# Cleanup when done
docker rmi test-standard:latest

# Or use --cleanup flag (auto-removes after tests pass)
./scripts/test-dockerfile.sh standard --cleanup

Troubleshooting

RUN Command Not Working

Symptom: Added commands to Dockerfile but they don't seem to execute or changes don't persist

Debugging Steps:

# 1. Check if command is in image history
docker history test-dev:latest --no-trunc | grep "your-command"

# 2. If found, check if changes actually exist in image
docker run --rm test-dev:latest ls -la /path/to/expected/files

# 3. If not found or layer shows CACHED, try cache bust
# Edit Dockerfile: add/change a comment in the RUN command

# 4. Test the command manually first (Pattern 5)
docker run --rm --user root test-dev:latest bash -c 'your-command'

Common Causes:

• Layer cache hiding your changes (try cache bust)
• Command fails silently in && chain (test commands individually)
• Wrong user (check if RUN runs as root but final image is non-root)
• Path issues (prototype in existing image first)

Files Missing in Final Image

Symptom: Files exist in build stage but not in final multi-stage image

Debugging Steps:

# 1. Build and inspect the intermediate stage
./scripts/build-local.sh your-stage test-stage:latest
docker run --rm test-stage:latest ls -la /expected/path

# 2. Compare to final image
docker run --rm test-dev:latest ls -la /expected/path

# 3. Check if symlinks are involved
docker run --rm test-stage:latest bash -c 'ls -la /path/ | grep "^l"'

Common Causes:

• Docker COPY follows symlinks (recreate them in final stage)
• COPY path doesn't include the files you expect
• Files in different location than you think
• ARG not declared in the stage where COPY happens

Build Shows CACHED But You Changed Dockerfile

Symptom: Modified Dockerfile but build output shows "CACHED" for your layer

Solutions:

# Option 1: Force cache bust with small change
# Add or modify a comment in the RUN command

# Option 2: Clear builder cache
docker builder prune -f

# Option 3: Verify your change is actually different
git diff Dockerfile  # Did the change actually save?

Why This Happens:

• Docker layer cache persists beyond image deletion
• Layer hash collision (rare but possible)
• Changes don't affect layer hash (comment-only changes)

Build is Slow

1. Check if layer cache is working: Look for "CACHED" in build output
2. Identify what changed: Compare to last build
3. Consider if rebuild is necessary: Can you test with existing image?
4. Last resort: This is normal for FROM scratch builds or major changes

Tests Failing After Dockerfile Changes

1. Test with existing image first: Isolate if it's a Dockerfile vs test issue
2. Check goss test validity: Can tests pass with old image?
3. Rebuild and test: ./scripts/test-dockerfile.sh

Image Doesn't Exist

# Check what you have
docker images | grep -E '(agentic|test-)'

# Build what you need
./scripts/test-dockerfile.sh dev  # Builds test-dev:latest

Cache Seems Corrupted

Symptoms:

• Build errors with "snapshot does not exist"
• Inconsistent build results
• Cache-related build failures

Only as last resort:

# Nuclear option: clear all cache and rebuild
docker system prune -f && docker builder prune -f

# Rebuild from scratch (will take several minutes)
./scripts/test-dockerfile.sh dev

Common Commands Reference

Building

# Build base standard target
./scripts/build-local.sh standard agentic-container:latest

# Build dev target
./scripts/build-local.sh dev agentic-container:dev

# Build specific stage
./scripts/build-local.sh python-stage python-only:latest

Testing

# Test base target (rebuilds)
./scripts/test-dockerfile.sh standard

# Test with cleanup
./scripts/test-dockerfile.sh standard --cleanup

# Test cookbook
./scripts/test-dockerfile.sh docs/cookbooks/python-cli/Dockerfile

# CI mode (use pre-built image)
./scripts/test-dockerfile.sh standard test-standard:latest

Interactive Debugging

# Using shell.sh (rebuilds)
./scripts/shell.sh standard

# Using existing image (no rebuild)
docker run --rm -it test-standard:latest bash

# Run a command in existing image
docker run --rm test-standard:latest python --version

Image Management

# List all images
docker images | grep -E '(agentic|test-)'

# Remove test images
docker rmi $(docker images -q 'test-*')

# Check image age
docker image inspect agentic-container:latest --format '{{.Created}}'

# Check Dockerfile age
ls -l Dockerfile

Debugging

# Inspect layer history
docker history test-dev:latest --no-trunc | grep "your-command"

# Test command manually as root
docker run --rm --user root test-dev:latest bash -c 'your-command'

# Interactive debugging session
docker run --rm -it --user root test-dev:latest bash

# Check if file exists in image
docker run --rm test-dev:latest ls -la /path/to/file

# Check for symlinks
docker run --rm test-dev:latest bash -c 'ls -la /path/ | grep "^l"'

# Compare files between stages
docker run --rm test-stage:latest ls -la /path
docker run --rm test-dev:latest ls -la /path

Cache Management

# Clear builder cache (recommended)
docker builder prune -f

# Clear all Docker cache (nuclear option)
docker system prune -f && docker builder prune -f

# Check builder cache size
docker system df

# Remove specific image (doesn't clear layers!)
docker rmi test-dev:latest

Decision Tree

Need to work with Docker image?
├─ Developing new Dockerfile commands?
│  ├─ Step 1: Prototype in existing image (Pattern 5)
│  ├─ Step 2: Add to Dockerfile
│  └─ Step 3: Test with ./scripts/test-dockerfile.sh
│
├─ RUN command not working as expected?
│  ├─ Check: docker history IMAGE | grep "command"
│  ├─ Test manually: docker run --user root IMAGE bash -c 'command'
│  └─ If CACHED: Add comment to bust cache
│
├─ Files missing in final image?
│  ├─ Build intermediate stage: ./scripts/build-local.sh STAGE
│  ├─ Inspect: docker run STAGE-IMAGE ls -la /path
│  └─ Check for symlinks: ls -la | grep "^l"
│
├─ Build shows CACHED but you changed Dockerfile?
│  ├─ Try: Modify a comment in the RUN command
│  ├─ Or: docker builder prune -f
│  └─ Verify: docker history IMAGE --no-trunc
│
├─ Making Dockerfile changes?
│  ├─ Yes → Run test-dockerfile.sh (will rebuild with cache)
│  └─ No ↓
│
├─ Making goss test changes only?
│  ├─ Yes → Use existing image with docker run + volume mounts
│  └─ No ↓
│
├─ Need interactive shell?
│  ├─ Image exists? → docker run -it IMAGE bash
│  └─ No image → ./scripts/test-dockerfile.sh TARGET
│
├─ Testing if something works?
│  ├─ Image exists? → docker run IMAGE your-test-command
│  └─ No image → ./scripts/test-dockerfile.sh TARGET
│
└─ Just want to build?
   └─ ./scripts/test-dockerfile.sh TARGET (preferred, includes tests)
   └─ Or: ./scripts/build-local.sh TARGET TAG (build only)

Key Takeaways

1. First run is always slower - Subsequent runs use layer cache
2. Scripts prioritize correctness - They rebuild to ensure clean state
3. Reuse images between script runs - Docker's layer cache + manual reuse
4. Don't fight the scripts - They're designed to be reliable, not fast
5. Layer cache is automatic - You get it for free with Docker
6. Manual cache management is rare - Only needed when truly broken
7. Test iterations don't need rebuilds - Mount goss files and test directly

When to Use This Skill

Invoke this skill when:

• User asks to build or test Docker images
• User asks how to iterate on Dockerfiles efficiently
• User complains about slow builds
• User asks about Docker cache
• User asks which script to use (build-local.sh vs test-dockerfile.sh vs shell.sh)
• User wants to test changes without rebuilding
• User needs to debug Docker images
• RUN command not working - Guide to test commands manually before adding to Dockerfile
• Files missing in final image - Debug multi-stage builds by inspecting intermediate stages
• Build shows CACHED but Dockerfile changed - Explain image vs layer cache difference
• Symlinks or special files not working - Docker COPY follows symlinks behavior
• Permission errors in containers - Test with --user root to debug
• Prototyping new features - Use existing images to test before modifying Dockerfile
• Multi-stage build debugging - Build and inspect intermediate stages