# Fly E2e Test > Deploy and test dspy-cli on Fly.io using local changes via temp git branch. Full integration testing with guaranteed cleanup. (project) Use when this capability is needed. - Skill: `tomevault-io/fly-e2e-test` (Agent Skill, multi-file: 2 files) - Install (CLI): `npx skillmds@latest add tomevault-io/fly-e2e-test` - Raw SKILL.md: https://api.skillmd.com/api/skills/tomevault-io/fly-e2e-test/raw - Safety review: pending (external: skill-scanner PASS, skillspector CAUTION) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: tomevault-io (https://skillmd.com/u/tomevault-io) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/tomevault-io/fly-e2e-test --- # Fly.io E2E Integration Test Skill Deploy a fresh dspy-cli project to Fly.io using your local code changes, run full integration tests (health, auth, LLM execution), and **guarantee cleanup** regardless of success or failure. ## ⚠️ CRITICAL RULES 1. **NEVER commit directly to main** - Always create a side branch first, even for small changes 2. **ALWAYS clean up** - Destroy Fly apps and delete temp branches, even if tests fail 3. **Use temp branches** - Name them `e2e-test/{timestamp}-{random}` for easy identification ## Prerequisites 1. **fly CLI**: Installed and authenticated (`fly auth whoami`) 2. **OPENAI_API_KEY**: In environment or `.env` file 3. **Git**: Clean working directory (stash uncommitted changes first) 4. **Git push access**: Ability to push to origin ## Quick Start Run each phase in a tmux session to enable output capture and cleanup tracking. ### Phase 1: Setup Environment ```bash # Create tmux session tmux new-session -d -s e2e-fly -c /Users/isaac/projects/dspy-cli # Set variables tmux send-keys -t e2e-fly 'export DSPY_CLI_DIR="/Users/isaac/projects/dspy-cli"' C-m tmux send-keys -t e2e-fly 'export TIMESTAMP=$(date +%s)' C-m tmux send-keys -t e2e-fly 'export RANDOM_SUFFIX=$(head -c 4 /dev/urandom | xxd -p)' C-m tmux send-keys -t e2e-fly 'export FLY_APP_NAME="dspy-e2e-${RANDOM_SUFFIX}"' C-m tmux send-keys -t e2e-fly 'export TEMP_BRANCH="e2e-test/${TIMESTAMP}-${RANDOM_SUFFIX}"' C-m # Source .env for OPENAI_API_KEY tmux send-keys -t e2e-fly 'set -a && source .env && set +a' C-m # Verify setup tmux send-keys -t e2e-fly 'echo "App: $FLY_APP_NAME Branch: $TEMP_BRANCH"' C-m ``` ### Phase 2: Pre-flight Checks ```bash # Verify fly CLI tmux send-keys -t e2e-fly 'fly version && fly auth whoami' C-m # Check for uncommitted changes (stash if needed) tmux send-keys -t e2e-fly 'git status --porcelain' C-m # Clean up any orphaned e2e resources tmux send-keys -t e2e-fly 'fly apps list 2>/dev/null | grep "dspy-e2e" || echo "No orphaned apps"' C-m ``` ### Phase 3: Create and Push Temp Branch ```bash tmux send-keys -t e2e-fly 'git checkout -b "$TEMP_BRANCH"' C-m tmux send-keys -t e2e-fly 'git push -u origin "$TEMP_BRANCH"' C-m ``` ### Phase 4: Create Test Project ```bash # Create temp directory tmux send-keys -t e2e-fly 'export TEST_DIR=$(mktemp -d) && echo "TEST_DIR=$TEST_DIR"' C-m # Create project (will prompt for API key confirmation - send Y) tmux send-keys -t e2e-fly 'uv run --directory "$DSPY_CLI_DIR" dspy-cli new fly-e2e-test --program-name qa_module --signature "question:str -> answer:str" --module-type Predict --model openai/gpt-4o-mini' C-m # When prompted "Proceed with this API key? [Y/n]:", send: tmux send-keys -t e2e-fly 'Y' C-m # Move project to temp dir (dspy-cli creates in current dir) tmux send-keys -t e2e-fly 'mv "$DSPY_CLI_DIR/fly-e2e-test" "$TEST_DIR/" && cd "$TEST_DIR/fly-e2e-test"' C-m ``` ### Phase 5: Modify for Git-Based dspy-cli ```bash # Update pyproject.toml to install dspy-cli from temp branch tmux send-keys -t e2e-fly 'sed -i.bak "s|\"dspy-cli\"|\"dspy-cli @ git+https://github.com/cmpnd-ai/dspy-cli.git@$TEMP_BRANCH\"|" pyproject.toml' C-m # IMPORTANT: Update Dockerfile to include git (required for git-based deps) # NOTE: This is an example dockerfile. There may be specific changes in a newer version of dspy-cli. Check the current Dockerfile and add the Git install line tmux send-keys -t e2e-fly 'cat > Dockerfile << '"'"'EOF'"'"' FROM python:3.11-slim ENV PYTHONDONTWRITEBYTECODE=1 ENV PYTHONUNBUFFERED=1 ENV XDG_CACHE_HOME=/tmp/.cache # Install git for fetching dspy-cli from git URL RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/* COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv COPY . . RUN uv sync --no-dev EXPOSE 8000 CMD ["uv", "run", "dspy-cli", "serve", "--host", "0.0.0.0", "--port", "8000", "--auth", "--no-reload"] EOF' C-m ``` ### Phase 6: Create fly.toml and Deploy ```bash # Create fly.toml tmux send-keys -t e2e-fly 'cat > fly.toml << EOF app = '"'"'$FLY_APP_NAME'"'"' primary_region = '"'"'ewr'"'"' [build] [http_service] internal_port = 8000 force_https = true auto_stop_machines = true auto_start_machines = true min_machines_running = 0 processes = ['"'"'app'"'"'] [[vm]] memory = '"'"'512mb'"'"' cpu_kind = '"'"'shared'"'"' cpus = 1 EOF' C-m # Create app tmux send-keys -t e2e-fly 'fly apps create "$FLY_APP_NAME" --org personal' C-m # Generate a random API key for testing tmux send-keys -t e2e-fly 'export DSPY_API_KEY_VALUE="test-e2e-$(head -c 8 /dev/urandom | xxd -p)"' C-m # Set secrets using fly secrets (required env vars for your app) # Add any additional env vars your project needs here tmux send-keys -t e2e-fly 'fly secrets set OPENAI_API_KEY="$OPENAI_API_KEY" DSPY_API_KEY="$DSPY_API_KEY_VALUE" --app "$FLY_APP_NAME"' C-m # Deploy (takes ~2-3 minutes) tmux send-keys -t e2e-fly 'fly deploy --app "$FLY_APP_NAME" --wait-timeout 300' C-m ``` ### Phase 7: Run Integration Tests ```bash tmux send-keys -t e2e-fly 'export FLY_APP_URL="https://$FLY_APP_NAME.fly.dev"' C-m # Test 1: Health Check tmux send-keys -t e2e-fly 'echo "=== Test 1: Health Check ===" && curl -s "$FLY_APP_URL/health"' C-m # Expected: {"status":"ok"} # Test 2: Auth Redirect (unauthenticated) tmux send-keys -t e2e-fly 'echo "=== Test 2: Auth Redirect ===" && curl -s -o /dev/null -w "HTTP: %{http_code}\n" "$FLY_APP_URL/programs"' C-m # Expected: HTTP: 303 # Test 3: Auth Success (authenticated) tmux send-keys -t e2e-fly 'echo "=== Test 3: Auth Success ===" && curl -s -H "Authorization: Bearer $DSPY_API_KEY_VALUE" "$FLY_APP_URL/programs"' C-m # Expected: {"programs":[{"name":"QaModulePredict",...}]} # Test 4: LLM Module Execution tmux send-keys -t e2e-fly 'echo "=== Test 4: LLM Execution ===" && curl -s -X POST -H "Authorization: Bearer $DSPY_API_KEY_VALUE" -H "Content-Type: application/json" -d '"'"'{"question": "What is 2+2?"}'"'"' "$FLY_APP_URL/QaModulePredict"' C-m # Expected: {"answer":"4"} (or similar) ``` ### Phase 8: Guaranteed Cleanup **ALWAYS run cleanup, even if tests fail:** ```bash # Destroy Fly app tmux send-keys -t e2e-fly 'fly apps destroy "$FLY_APP_NAME" --yes' C-m # Delete remote branch tmux send-keys -t e2e-fly 'git -C "$DSPY_CLI_DIR" push origin --delete "$TEMP_BRANCH"' C-m # Return to main and delete local branch tmux send-keys -t e2e-fly 'git -C "$DSPY_CLI_DIR" checkout main' C-m tmux send-keys -t e2e-fly 'git -C "$DSPY_CLI_DIR" branch -D "$TEMP_BRANCH"' C-m # Remove temp directory tmux send-keys -t e2e-fly 'rm -rf "$TEST_DIR"' C-m # Kill tmux session tmux kill-session -t e2e-fly ``` ## Verification Checklist | Test | Expected Result | |------|-----------------| | Health Check | `{"status":"ok"}` | | Auth Redirect (no auth) | HTTP 303 | | Auth Success (Bearer token) | JSON with `QaModulePredict` | | LLM Execution | JSON with `"answer"` field | ## Cleanup Verification After running cleanup, verify: ```bash # No orphaned Fly apps fly apps list | grep "dspy-e2e" || echo "Clean" # No orphaned branches git branch -r | grep "e2e-test/" || echo "Clean" ``` ## Troubleshooting ### Deploy fails with "Git executable not found" The Dockerfile must include git installation. Ensure the Dockerfile has: ```dockerfile RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/* ``` ### pyproject.toml sed command doesn't expand variables Use double quotes for the sed command, not single quotes: ```bash sed -i.bak "s|...|...@$TEMP_BRANCH\"|" pyproject.toml # Correct sed -i.bak 's|...|...|' pyproject.toml # Won't expand $TEMP_BRANCH ``` ### Project created in wrong directory `dspy-cli new` creates projects relative to the current working directory, not where it's run from. Move the project after creation: ```bash mv "$DSPY_CLI_DIR/fly-e2e-test" "$TEST_DIR/" ``` ### Cleanup fails If any cleanup step fails, run them individually: ```bash fly apps destroy "dspy-e2e-XXXX" --yes git push origin --delete "e2e-test/XXXX" git checkout main git branch -D "e2e-test/XXXX" ``` ### App crashes due to missing environment variables Use `fly secrets` to set any required env vars. Check the app logs to see which vars are missing: ```bash # View logs to find missing env vars fly logs --app "$FLY_APP_NAME" --no-tail # Set additional secrets as needed fly secrets set VAR_NAME="value" ANOTHER_VAR="value" --app "$FLY_APP_NAME" # List current secrets fly secrets list --app "$FLY_APP_NAME" ``` Common env vars that might be needed: - `OPENAI_API_KEY` - Required for OpenAI models - `DSPY_API_KEY` - Required when `--auth` is enabled - Project-specific vars (check your gateway's `setup()` method) ## Multi-Layer Cleanup Protection 1. **Unique naming**: `dspy-e2e-{random}` prevents conflicts 2. **Pre-test orphan cleanup**: Removes stale resources before starting 3. **tmux session**: Enables output capture and manual recovery 4. **Explicit cleanup phase**: Always runs after tests 5. **Verification commands**: Confirm cleanup succeeded --- > Source: [cmpnd-ai/dspy-cli](https://github.com/cmpnd-ai/dspy-cli) — distributed by [TomeVault](https://tomevault.io).