Pr Test

Manual E2E Test

Test a PR/branch end-to-end by building the full platform, interacting via browser and API, capturing screenshots, and reporting results.

Changelog 2.2.0 — auth flow updated for Better Auth (Supabase signup is gone), env-setup gaps closed, a proven Playwright fallback for agent-browser, a billing-test trap that produces false passes, safer process cleanup, and a mock-provider pattern for deterministic $0 testing. Learned on #14206 — see the evidence comment. 2.2.1 — corrects two claims from 2.2.0 that didn't survive live-stack verification (JWKS does not rotate on a frontend restart; the local Postgres port is 5432, not 54322) and hardens the auth setup (explicit password-length/allowlist/rate-limit failure modes, fail-fast on an empty token, password kept out of process args).

Critical Requirements

These are NON-NEGOTIABLE. Every test run MUST satisfy ALL the following:

1. Screenshots at Every Step

• Take a screenshot at EVERY significant test step — not just at the end
• Every test scenario MUST have at least one BEFORE and one AFTER screenshot
• Name screenshots sequentially: {NN}-{action}-{state}.png (e.g., 01-credits-before.png, 02-credits-after.png)
• If a screenshot is missing for a scenario, the test is INCOMPLETE — go back and take it

2. Screenshots MUST Be Posted to PR

• Push ALL screenshots to a temp branch test-screenshots/pr-{N}
• Post a PR comment with ALL screenshots embedded inline using GitHub raw URLs
• This is NOT optional — every test run MUST end with a PR comment containing screenshots
• If screenshot upload fails, retry. If it still fails, list failed files and require manual drag-and-drop/paste attachment in the PR comment

3. State Verification with Before/After Evidence

• For EVERY state-changing operation (API call, user action), capture the state BEFORE and AFTER
• Log the actual API response values (e.g., credits_before=100, credits_after=95)
• Screenshot MUST show the relevant UI state change
• Compare expected vs actual values explicitly — do not just eyeball it

4. Negative Test Cases Are Mandatory

• Test at least ONE negative case per feature (e.g., insufficient credits, invalid input, unauthorized access)
• Verify error messages are user-friendly and accurate
• Verify the system state did NOT change after a rejected operation

5. Test Report Must Include Full Evidence

Each test scenario in the report MUST have:

• Steps: What was done (exact commands or UI actions)
• Expected: What should happen
• Actual: What actually happened
• API Evidence: Before/after API response values for state-changing operations
• Screenshot Evidence: Before/after screenshots with explanations

State Manipulation for Realistic Testing

Billing-test trap — this one produces a false pass, not a visible failure. LLM block cost filters key on the platform-owned credential id. A run made with the test user's own API key bills nothing by design, so a credits before/after assertion silently passes on zero deltas either way. Any test that verifies credit reconciliation MUST use the system credential, not a user-supplied key. Related: Ollama block entries are configured with an explicit $0 run-based cost (BlockCostType.RUN, cost_amount=0 in block_cost_config.py), not a token-metered one — so pre/post-flight cost deltas are always 0 for them regardless of credential. Never use an Ollama model to test credit reconciliation; pick any hosted model billed through the system credential instead.

When testing features that depend on specific states (rate limits, credits, quotas):

1. Use Redis CLI to set counters directly:

   # Find the Redis container
   REDIS_CONTAINER=$(docker ps --format '{{.Names}}' | grep redis | head -1)
   # Set a key with expiry
   docker exec $REDIS_CONTAINER redis-cli SET key value EX ttl
   # Example: Set rate limit counter to near-limit
   docker exec $REDIS_CONTAINER redis-cli SET "rate_limit:user:$PR_TEST_USER_EMAIL" 99 EX 3600
   # Example: Check current value
   docker exec $REDIS_CONTAINER redis-cli GET "rate_limit:user:$PR_TEST_USER_EMAIL"
   

2. Use API calls to check before/after state:

   # BEFORE: Record current state
   BEFORE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
   echo "Credits BEFORE: $BEFORE"
   
   # Perform the action...
   
   # AFTER: Record new state and compare
   AFTER=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
   echo "Credits AFTER: $AFTER"
   echo "Delta: $(( BEFORE - AFTER ))"
   

3. Take screenshots BEFORE and AFTER state changes — the UI must reflect the backend state change

4. Never rely on mocked/injected browser state — always use real backend state. Do NOT use agent-browser eval to fake UI state. The backend must be the source of truth.

5. Use direct DB queries when needed:

   # Query via Supabase's PostgREST or docker exec into the DB
   docker exec supabase-db psql -U supabase_admin -d postgres -c "SELECT credits FROM user_credits WHERE user_id = '...';"
   

6. After every API test, verify the state change actually persisted:

   # Example: After a credits purchase, verify DB matches API
   API_CREDITS=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/credits | jq '.credits')
   DB_CREDITS=$(docker exec supabase-db psql -U supabase_admin -d postgres -t -c "SELECT credits FROM user_credits WHERE user_id = '...';" | tr -d ' ')
   [ "$API_CREDITS" = "$DB_CREDITS" ] && echo "CONSISTENT" || echo "MISMATCH: API=$API_CREDITS DB=$DB_CREDITS"
   

Arguments

• $ARGUMENTS — worktree path (e.g. $REPO_ROOT) or PR number
• If --fix flag is present, auto-fix bugs found and push fixes (like pr-address loop)

Step 0: Resolve the target

# If argument is a PR number, find its worktree
gh pr view {N} --json headRefName --jq '.headRefName'
# If argument is a path, use it directly

Determine:

• REPO_ROOT — the root repo directory: git -C "$WORKTREE_PATH" worktree list | head -1 | awk '{print $1}' (or git rev-parse --show-toplevel if not a worktree)
• WORKTREE_PATH — the worktree directory
• PLATFORM_DIR — $WORKTREE_PATH/autogpt_platform
• BACKEND_DIR — $PLATFORM_DIR/backend
• FRONTEND_DIR — $PLATFORM_DIR/frontend
• PR_NUMBER — the PR number (from gh pr list --head $(git branch --show-current))
• PR_TITLE — the PR title, slugified (e.g. "Add copilot permissions" → "add-copilot-permissions")
• RESULTS_DIR — $REPO_ROOT/test-results/PR-{PR_NUMBER}-{slugified-title}

Create the results directory:

PR_NUMBER=$(cd $WORKTREE_PATH && gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT --json number --jq '.[0].number')
PR_TITLE=$(cd $WORKTREE_PATH && gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT --json title --jq '.[0].title' | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/--*/-/g' | sed 's/^-//;s/-$//' | head -c 50)
RESULTS_DIR="$REPO_ROOT/test-results/PR-${PR_NUMBER}-${PR_TITLE}"
mkdir -p $RESULTS_DIR

Test user credentials — required to log into the UI or call authenticated APIs.

NEVER hardcode these in this SKILL, a PR comment, a screenshot, or any committed file. Sources, in priority order:

1. Env vars (CI / preconfigured local shell): $PR_TEST_USER_EMAIL + $PR_TEST_USER_PASSWORD. If both are set, use them.
2. Interactive prompt (everything else, including dev-preview runs): if the env vars are not set, ASK the user at the start of the run — e.g. "I need test-user credentials for this run; paste the email and password." Hold them in shell vars only for the duration of the run. Do not echo, log, or write them to disk.

Acquire the variables — env first, prompt if missing — and only then lock them in:

# 1. Prefer env vars (CI / preconfigured shell). Prompt only for the
#    specific var that is unset so an already-exported credential is
#    not overwritten by the prompt when only the other one is missing.
if [ -z "${PR_TEST_USER_EMAIL:-}" ] || [ -z "${PR_TEST_USER_PASSWORD:-}" ]; then
  echo "Test user credentials required for this run."
  if [ -z "${PR_TEST_USER_EMAIL:-}" ]; then
    read -r -p   "Email:    " PR_TEST_USER_EMAIL
  fi
  if [ -z "${PR_TEST_USER_PASSWORD:-}" ]; then
    read -r -s -p "Password: " PR_TEST_USER_PASSWORD
    echo
  fi
  export PR_TEST_USER_EMAIL PR_TEST_USER_PASSWORD
fi

# 2. Lock them in — fail loudly if either is STILL unset (e.g. the user
#    pressed Enter on an empty prompt). The error message names the var so
#    the agent / operator knows what to fix.
: "${PR_TEST_USER_EMAIL:?PR_TEST_USER_EMAIL is empty after env+prompt — supply a value before re-running}"
: "${PR_TEST_USER_PASSWORD:?PR_TEST_USER_PASSWORD is empty after env+prompt — supply a value before re-running}"

For local docker-compose runs, a fresh dev user is created on first call to the signup snippet below. For dev-preview runs, the test user lives in the project's hosted auth backend — ask the user for the current valid credentials each session (the previously-shared test@test.com test account was disabled on 2026-05-23 after its credentials leaked into this very SKILL — do NOT re-introduce a default). PR_TEST_USER_PASSWORD should always be a throwaway/test-only credential, never a real account's password — the auth requests in 3h go over plain HTTP on localhost:3000 for local runs, which has no transport encryption. If a dev-preview run's target isn't on localhost, confirm it's HTTPS before sending credentials to it.

Step 1: Understand the PR

Before testing, understand what changed:

cd $WORKTREE_PATH

# Read PR description to understand the WHY
gh pr view {N} --json body --jq '.body'

git log --oneline dev..HEAD | head -20
git diff dev --stat

Read the PR description (Why / What / How) and changed files to understand: 0. Why does this PR exist? What problem does it solve?

1. What feature/fix does this PR implement?
2. How does it work? What's the approach?
3. What components are affected? (backend, frontend, copilot, executor, etc.)
4. What are the key user-facing behaviors to test?

Step 2: Write test scenarios

Based on the PR analysis, write a test plan to $RESULTS_DIR/test-plan.md:

# Test Plan: PR #{N} — {title}

## Scenarios
1. [Scenario name] — [what to verify]
2. ...

## API Tests (if applicable)
1. [Endpoint] — [expected behavior]
   - Before state: [what to check before]
   - After state: [what to verify changed]

## UI Tests (if applicable)
1. [Page/component] — [interaction to test]
   - Screenshot before: [what to capture]
   - Screenshot after: [what to capture]

## Negative Tests (REQUIRED — at least one per feature)
1. [What should NOT happen] — [how to trigger it]
   - Expected error: [what error message/code]
   - State unchanged: [what to verify did NOT change]

Be critical — include edge cases, error paths, and security checks. Every scenario MUST specify what screenshots to take and what state to verify.

Step 3.0: Claim the testing lock (coordinate parallel agents)

Multiple worktrees share the same host — Docker infra (postgres, redis, clamav), app ports (3000/8006/…), and the test user. Two agents running /pr-test concurrently will corrupt each other's state (connection-pool exhaustion, port binds failing silently, cross-test assertions). Use the root-worktree lock file to take turns.

Lock file contract

Path (always the root worktree so all siblings see it): $REPO_ROOT/.ign.testing.lock

Body (one key=value per line):

holder=<pr-XXXXX-purpose>
pid=<pid-or-"self">
started=<iso8601>
heartbeat=<iso8601, updated every ~2 min>
worktree=<full path>
branch=<branch name>
intent=<one-line description + rough duration>

Claim

LOCK=$REPO_ROOT/.ign.testing.lock
NOW=$(date -u +%Y-%m-%dT%H:%MZ)
STALE_AFTER_MIN=5

if [ -f "$LOCK" ]; then
  HB=$(grep '^heartbeat=' "$LOCK" | cut -d= -f2)
  HB_EPOCH=$(date -j -f '%Y-%m-%dT%H:%MZ' "$HB" +%s 2>/dev/null || date -d "$HB" +%s 2>/dev/null || echo 0)
  AGE_MIN=$(( ( $(date -u +%s) - HB_EPOCH ) / 60 ))
  if [ "$AGE_MIN" -gt "$STALE_AFTER_MIN" ]; then
    echo "WARN: stale lock (${AGE_MIN}m old) — reclaiming"
    cat "$LOCK" | sed 's/^/  stale: /'
  else
    echo "Another agent holds the lock:"; cat "$LOCK"
    echo "Wait until released or resume after $((STALE_AFTER_MIN - AGE_MIN))m."
    exit 1
  fi
fi

cat > "$LOCK" <<EOF
holder=pr-${PR_NUMBER}-e2e
pid=self
started=$NOW
heartbeat=$NOW
worktree=$WORKTREE_PATH
branch=$(cd $WORKTREE_PATH && git branch --show-current)
intent=E2E test PR #${PR_NUMBER}, native mode, ~60min
EOF
echo "Lock claimed"

Heartbeat (MUST run in background during the whole test)

Without a heartbeat a crashed agent keeps the lock forever. Run this as a background process right after claim:

(while true; do
   sleep 120
   [ -f "$LOCK" ] || exit 0   # lock released → exit heartbeat
   perl -i -pe "s/^heartbeat=.*/heartbeat=$(date -u +%Y-%m-%dT%H:%MZ)/" "$LOCK"
 done) &
HEARTBEAT_PID=$!
echo "$HEARTBEAT_PID" > /tmp/pr-test-heartbeat.pid

Release (always — even on failure)

kill "$HEARTBEAT_PID" 2>/dev/null
rm -f "$LOCK" /tmp/pr-test-heartbeat.pid
echo "$(date -u +%Y-%m-%dT%H:%MZ) [pr-${PR_NUMBER}] released lock" \
    >> $REPO_ROOT/.ign.testing.log

Use a trap so release runs even on exit 1:

trap 'kill "$HEARTBEAT_PID" 2>/dev/null; rm -f "$LOCK"' EXIT INT TERM

Release the lock AS SOON AS the test run is done

The lock guards test execution, not app lifecycle. Once Step 5 (record results) and Step 6 (post PR comment) are complete, release the lock IMMEDIATELY — even if:

• The native poetry run app / pnpm dev processes are still running so the user can keep poking at the app manually.
• You're leaving docker containers up.
• You're tailing logs for a minute or two.

Keeping the lock held past the test run is the single most common way /pr-test stalls other agents. The app staying up is orthogonal to the lock; don't conflate them. Sibling worktrees running their own /pr-test will kill the stray processes and free the ports themselves (Step 3c/3e-native handle that) — they just need the lock file gone.

Concretely, the sequence at the end of every /pr-test run (success or failure) is:

# 1. Write the final report + post PR comment — done above in Step 5/6.
# 2. Release the lock right now, even if the app is still up.
kill "$HEARTBEAT_PID" 2>/dev/null
rm -f "$LOCK" /tmp/pr-test-heartbeat.pid
echo "$(date -u +%Y-%m-%dT%H:%MZ) [pr-${PR_NUMBER}] released lock (app may still be running)" \
    >> $REPO_ROOT/.ign.testing.log
# 3. Optionally leave the app running and note it so the user knows:
echo "Native stack still running on :3000 / :8006 for manual poking. Kill with:"
echo "  pkill -9 -f 'poetry run app'; pkill -9 -f 'next-server|next dev'"

If a sibling agent's /pr-test needs to take over, it'll do the kill+rebuild dance from Step 3c/3e-native on its own — your only job is to not hold the lock file past the end of your test.

Shared status log

$REPO_ROOT/.ign.testing.log is an append-only channel any agent can read/write. Use it for "I'm waiting", "I'm done, resources free", or post-run notes:

echo "$(date -u +%Y-%m-%dT%H:%MZ) [pr-${PR_NUMBER}] <message>" \
    >> $REPO_ROOT/.ign.testing.log

Step 3: Environment setup

3a. Copy .env files from the root worktree

The root worktree ($REPO_ROOT) has the canonical .env files with all API keys. Copy them to the target worktree:

# CRITICAL: .env files are NOT checked into git. They must be copied manually.
cp $REPO_ROOT/autogpt_platform/.env $PLATFORM_DIR/.env
cp $REPO_ROOT/autogpt_platform/backend/.env $BACKEND_DIR/.env
cp $REPO_ROOT/autogpt_platform/frontend/.env $FRONTEND_DIR/.env

A copy from the root worktree is no longer sufficient on a recent dev — auth moved from Supabase to Better Auth (see 3h) and two vars are easy to miss because nothing fails loudly without them, it just 401s later:

• $BACKEND_DIR/.env needs JWT_JWKS_URL — the Better Auth JWKS endpoint the backend verifies tokens against. The localhost:3000 value below is for native mode only. In docker mode it's harmless to have it in .env because docker-compose.platform.yml overrides it with the Compose-reachable http://frontend:3000/api/auth/jwks — but if you ever run the backend against this .env value directly (bypassing Compose), a localhost value inside a container resolves to itself, not the frontend, and every call 401s with no other symptom.
• $FRONTEND_DIR/.env needs its own DATABASE_URL — Better Auth runs inside the Next.js app and talks to Postgres directly, it does not go through the backend. Derive it from $BACKEND_DIR/.env's DB_USER / DB_PASS / DB_PORT / DB_NAME rather than copying frontend/.env.default's placeholder verbatim — the placeholder happens to match the stock local defaults, but if backend/.env's DB_PASS was ever customized (rotated secret, non-default port), copying the placeholder silently points Better Auth at the wrong database instead of the one the rest of the stack actually uses.

# [ -n ... ], not grep -q alone — a present-but-empty JWT_JWKS_URL= would
# otherwise be treated as "already set" and skip the fallback, leaving the
# backend without a JWKS endpoint to verify tokens against.
[ -n "$(grep '^JWT_JWKS_URL=' $BACKEND_DIR/.env | cut -d= -f2-)" ] || echo "JWT_JWKS_URL=http://localhost:3000/api/auth/jwks" >> $BACKEND_DIR/.env  # native mode only — see note above

if [ -n "$(grep '^DATABASE_URL=' $FRONTEND_DIR/.env | cut -d= -f2-)" ]; then
  # grep -q alone matches a present-but-empty DATABASE_URL= too, which would
  # otherwise skip derivation and leave Better Auth pointed at nothing.
  echo "Frontend DATABASE_URL: already set (not touching it)"
else
  # cut -f2 (not -f2-) truncates any value containing '=' (base64 secrets do); -f2- keeps the rest.
  DB_USER=$(grep '^DB_USER=' $BACKEND_DIR/.env | cut -d= -f2-)
  DB_PASS=$(grep '^DB_PASS=' $BACKEND_DIR/.env | cut -d= -f2-)
  DB_PORT=$(grep '^DB_PORT=' $BACKEND_DIR/.env | cut -d= -f2-)
  DB_NAME=$(grep '^DB_NAME=' $BACKEND_DIR/.env | cut -d= -f2-)
  : "${DB_USER:?}" "${DB_PASS:?}" "${DB_PORT:?}" "${DB_NAME:?}"  # fail loudly, not with a silently-empty URL
  # Percent-encode user/pass — a raw '@', '#', '?', '%', or ':' in either would
  # otherwise be misparsed as URL structure instead of credential content.
  # Via env vars, not `jq --arg`, which would put DB_PASS in the process arglist.
  DB_USER_ENC=$(DB_USER_VAL="$DB_USER" jq -rn '$ENV.DB_USER_VAL|@uri')
  DB_PASS_ENC=$(DB_PASS_VAL="$DB_PASS" jq -rn '$ENV.DB_PASS_VAL|@uri')
  echo "DATABASE_URL=postgresql://${DB_USER_ENC}:${DB_PASS_ENC}@localhost:${DB_PORT}/${DB_NAME}" >> $FRONTEND_DIR/.env
  # Reconstructed, not regex-redacted — a password containing '@' would otherwise
  # leak its tail past a naive "redact up to the first @" pattern.
  echo "Frontend DATABASE_URL: postgresql://${DB_USER_ENC}:***@localhost:${DB_PORT}/${DB_NAME}"
fi

3b. Configure copilot authentication

The copilot needs an LLM API to function. Two approaches (try subscription first):

Option 1: Subscription mode (preferred — uses your Claude Max/Pro subscription)

The claude_agent_sdk Python package bundles its own Claude CLI binary — no need to install @anthropic-ai/claude-code via npm. The backend auto-provisions credentials from environment variables on startup.

Run the helper script to extract tokens from your host and auto-update backend/.env (works on macOS, Linux, and Windows/WSL):

# Extracts OAuth tokens and writes CLAUDE_CODE_OAUTH_TOKEN + CLAUDE_CODE_REFRESH_TOKEN into .env
bash $BACKEND_DIR/scripts/refresh_claude_token.sh --env-file $BACKEND_DIR/.env

How it works: The script reads the OAuth token from:

• macOS: system keychain ("Claude Code-credentials")
• Linux/WSL: ~/.claude/.credentials.json
• Windows: %APPDATA%/claude/.credentials.json

It sets CLAUDE_CODE_OAUTH_TOKEN, CLAUDE_CODE_REFRESH_TOKEN, and CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=true in the .env file. On container startup, the backend auto-provisions ~/.claude/.credentials.json inside the container from these env vars. The SDK's bundled CLI then authenticates using that file. No claude login, no npm install needed.

Note: The OAuth token expires (~24h). If copilot returns auth errors, re-run the script and restart: $BACKEND_DIR/scripts/refresh_claude_token.sh --env-file $BACKEND_DIR/.env && docker compose up -d copilot_executor

Option 2: OpenRouter API key mode (fallback)

If subscription mode doesn't work, switch to API key mode using OpenRouter:

# In $BACKEND_DIR/.env, ensure these are set:
CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=false
CHAT_API_KEY=<value of OPEN_ROUTER_API_KEY from the same .env>
CHAT_BASE_URL=https://openrouter.ai/api/v1
CHAT_USE_CLAUDE_AGENT_SDK=true

Use sed to update these values:

ORKEY=$(grep "^OPEN_ROUTER_API_KEY=" $BACKEND_DIR/.env | cut -d= -f2)
[ -n "$ORKEY" ] || { echo "ERROR: OPEN_ROUTER_API_KEY is missing in $BACKEND_DIR/.env"; exit 1; }
perl -i -pe 's/CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=true/CHAT_USE_CLAUDE_CODE_SUBSCRIPTION=false/' $BACKEND_DIR/.env
# Add or update CHAT_API_KEY and CHAT_BASE_URL
grep -q "^CHAT_API_KEY=" $BACKEND_DIR/.env && perl -i -pe "s|^CHAT_API_KEY=.*|CHAT_API_KEY=$ORKEY|" $BACKEND_DIR/.env || echo "CHAT_API_KEY=$ORKEY" >> $BACKEND_DIR/.env
grep -q "^CHAT_BASE_URL=" $BACKEND_DIR/.env && perl -i -pe 's|^CHAT_BASE_URL=.*|CHAT_BASE_URL=https://openrouter.ai/api/v1|' $BACKEND_DIR/.env || echo "CHAT_BASE_URL=https://openrouter.ai/api/v1" >> $BACKEND_DIR/.env

3c. Stop conflicting containers

# Stop any running app containers (keep infra: supabase, redis, rabbitmq, clamav)
docker ps --format "{{.Names}}" | grep -E "rest_server|executor|copilot|websocket|database_manager|scheduler|notification|frontend|migrate" | while read name; do
  docker stop "$name" 2>/dev/null
done

Native mode also: when running the app natively (see 3e-native), kill any stray host processes and free the app ports before starting — otherwise poetry run app and pnpm dev will fail to bind.

Kill by port, not by broad process pattern. A pattern-based pkill -f "python.*backend" (or anything matching by worktree cwd) is too coarse on a host running several worktrees — it has taken out the frontend and a mock server sitting on other ports along with the intended backend process. Target the pid actually holding each port instead — this only kills whoever is bound to that specific port, which is narrower than a pattern match, but it is not worktree isolation: if a sibling worktree's own dev server happens to be using one of these ports (e.g. its frontend also on :3000), this kills that too. lsof tells you who holds the port, not who owns it — a docker-proxy pid there means a compose stack owns it.

# Free app ports one at a time — errors per port are ignored (port may simply
# be unused). `xargs -r` is GNU-only (macOS xargs rejects -r); the `[ -n ]`
# guard below is the portable equivalent.
for port in 3000 8006 8001 8002 8005 8008; do
  pids=$(lsof -ti :$port -sTCP:LISTEN 2>/dev/null)
  [ -n "$pids" ] && kill -9 $pids 2>/dev/null || true
done

3e-native. Run the app natively (PREFERRED for iterative dev)

Native mode runs infra (postgres, supabase, redis, rabbitmq, clamav) in docker but runs the backend and frontend directly on the host. This avoids the 3-8 minute docker compose build cycle on every backend change — code edits are picked up on process restart (seconds) instead of a full image rebuild.

When to prefer native mode (default for this skill):

• Iterative dev/debug loops where you're editing backend or frontend code between test runs
• Any PR that touches Python/TS source but not Dockerfiles, compose config, or infra images
• Fast repro of a failing scenario — restart poetry run app in a couple of seconds

When to prefer docker mode (3e fallback):

• Testing changes to Dockerfile, docker-compose.yml, or base images
• Production-parity smoke tests (exact container env, networking, volumes)
• CI-equivalent runs where you need the exact image that'll ship

Note on 3b (copilot auth): no npm install anywhere. poetry install pulls in claude_agent_sdk, which ships its own Claude CLI binary — available on PATH whenever you run commands via poetry run (native) OR whenever the copilot_executor container is built from its Poetry lockfile (docker). The OAuth token extraction still applies (same refresh_claude_token.sh call).

Preamble: before starting native, run the kill-stray + free-ports block from 3c's "Native mode also" subsection.

1. Start infra only (one-time per session):

cd $PLATFORM_DIR && docker compose --profile local up deps --detach --remove-orphans --build

This brings up postgres/supabase/redis/rabbitmq/clamav and skips all app services.

2. Start the backend natively:

cd $BACKEND_DIR && (poetry run app 2>&1 | tee .ign.application.logs) &

poetry run app spawns all app subprocesses — rest_server, executor, copilot_executor, websocket, scheduler, notification_server, database_manager — inside ONE parent process. No separate containers, no separate terminals. The .ign.application.logs prefix is already gitignored.

3. Wait for the backend on :8006 BEFORE starting the frontend. This ordering matters — the frontend's pnpm dev startup invokes generate-api-queries, which fetches /openapi.json from the backend. If the backend isn't listening yet, pnpm dev fails immediately.

for i in $(seq 1 60); do
  if [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8006/docs 2>/dev/null)" = "200" ]; then
    echo "Backend ready"
    break
  fi
  sleep 2
done

4. Start the frontend natively:

cd $FRONTEND_DIR && (pnpm dev 2>&1 | tee .ign.frontend.logs) &

5. Wait for the frontend on :3000:

for i in $(seq 1 60); do
  if [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:3000 2>/dev/null)" = "200" ]; then
    echo "Frontend ready"
    break
  fi
  sleep 2
done

Once both are up, skip 3e/3f and go straight to 3g/3h (feature flags / test user creation).

3e. Build and start (docker — fallback)

cd $PLATFORM_DIR && docker compose build --no-cache 2>&1 | tail -20
if [ ${PIPESTATUS[0]} -ne 0 ]; then echo "ERROR: Docker build failed"; exit 1; fi

cd $PLATFORM_DIR && docker compose up -d 2>&1 | tail -20
if [ ${PIPESTATUS[0]} -ne 0 ]; then echo "ERROR: Docker compose up failed"; exit 1; fi

Note: If the container appears to be running old code (e.g. missing PR changes), use docker compose build --no-cache to force a full rebuild. Docker BuildKit may sometimes reuse cached COPY layers from a previous build on a different branch.

Expected time: 3-8 minutes for build, 5-10 minutes with --no-cache.

3f. Wait for services to be ready

# Poll until backend and frontend respond
for i in $(seq 1 60); do
  BACKEND=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8006/docs 2>/dev/null)
  FRONTEND=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3000 2>/dev/null)
  if [ "$BACKEND" = "200" ] && [ "$FRONTEND" = "200" ]; then
    echo "Services ready"
    break
  fi
  sleep 5
done

3h. Create test user and get auth token

The platform moved off Supabase auth to Better Auth, embedded in the Next.js app at /api/auth/*. Signup and sign-in both go through the frontend now, not Kong on :8000 — and /api/auth/token mints a backend-API JWT from a session cookie, it does not accept credentials directly, so sign-in has to happen first to get that cookie.

Better Auth's default minimum password length is 12 characters — shorter values fail signup with PASSWORD_TOO_SHORT and every step below degrades silently into an empty token unless you check for it.

COOKIE_JAR=$(mktemp)
trap 'rm -f "$COOKIE_JAR"' EXIT  # cleans up on early exit too, not just the happy path
# Via env vars, not `jq --arg`, which would put the password in the process arglist.
AUTH_PAYLOAD=$(PR_TEST_USER_EMAIL="$PR_TEST_USER_EMAIL" PR_TEST_USER_PASSWORD="$PR_TEST_USER_PASSWORD" \
  jq -nc '{email:$ENV.PR_TEST_USER_EMAIL,password:$ENV.PR_TEST_USER_PASSWORD,name:"PR Test User"}')

# Signup (idempotent — a real error body means "already exists" only if you
# check; -d passes the payload as an argument, which leaks the password into
# process listings, so pipe it through stdin with --data-binary @- instead.
# --noproxy guards against an inherited proxy env var routing the password
# through a proxy. --max-time bounds it — without one, a server that accepts
# the connection but never responds hangs setup indefinitely instead of
# reaching the empty-$TOKEN failure check below).
SIGNUP_RESULT=$(printf '%s' "$AUTH_PAYLOAD" | curl -s --max-time 15 --noproxy localhost,127.0.0.1,::1 -X POST 'http://localhost:3000/api/auth/sign-up/email' \
  -H 'Content-Type: application/json' --data-binary @-)
echo "$SIGNUP_RESULT" | grep -qi '"code"' && echo "Signup: $SIGNUP_RESULT"  # log it — "already exists" and "password too short" look identical downstream otherwise

# Sign in — sets the better-auth.session_token cookie in $COOKIE_JAR.
# Capture the body: a failure here (e.g. account exists with a different
# password) otherwise only shows up as an empty $TOKEN with no explanation.
SIGNIN_RESULT=$(printf '%s' "$AUTH_PAYLOAD" | curl -s --max-time 15 --noproxy localhost,127.0.0.1,::1 -c "$COOKIE_JAR" -X POST 'http://localhost:3000/api/auth/sign-in/email' \
  -H 'Content-Type: application/json' --data-binary @-)
echo "$SIGNIN_RESULT" | grep -qi '"code"' && echo "Sign-in: $SIGNIN_RESULT"

# Mint a backend-API JWT from the session cookie
TOKEN=$(curl -s -b "$COOKIE_JAR" 'http://localhost:3000/api/auth/token' | jq -r '.token // ""')
[ -n "$TOKEN" ] || { echo "ERROR: auth setup failed — TOKEN is empty. Check password length (min 12 chars), AUTH_ALLOW_NEW_ACCOUNTS, AUTH_SIGNUP_ALLOWLIST, and rate limiting on /api/auth/*."; exit 1; }

Use this token for ALL API calls:

curl -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/...

3i. Disable onboarding for test user

The frontend redirects to /onboarding when the ONBOARDING_COMPLETE step is not in completedSteps. Mark it complete via the backend API so every browser test lands on the real feature UI:

ONBOARDING_RESULT=$(curl -s --max-time 30 -X POST \
  "http://localhost:8006/api/onboarding/step?step=ONBOARDING_COMPLETE" \
  -H "Authorization: Bearer $TOKEN")
echo "Onboarding bypass: $ONBOARDING_RESULT"

# Verify it took effect -s --max-time 30 \
  "http://localhost:8006/api/onboarding/completed" \
  -H "Authorization: Bearer $TOKEN" | jq -r '.is_completed')
echo "Onboarding completed: $ONBOARDING_STATUS"
if [ "$ONBOARDING_STATUS" != "true" ]; then
  echo "ERROR: onboarding bypass failed — browser tests will hit /onboarding instead of the target feature. Investigate before proceeding."
  exit 1
fi

Step 4: Run tests

Service ports reference

API testing

Use curl with the auth token for backend API tests. For EVERY API call that changes state, record before/after values:

# Example: List agents
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/graphs | jq . | head -20

# Example: Create an agent
curl -s -X POST http://localhost:8006/api/graphs \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{...}' | jq .

# Example: Run an agent
curl -s -X POST "http://localhost:8006/api/graphs/{graph_id}/execute" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"data": {...}}'

# Example: Get execution results
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://localhost:8006/api/graphs/{graph_id}/executions/{exec_id}" | jq .

State verification pattern (use for EVERY state-changing API call):

# 1. Record BEFORE state
BEFORE_STATE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/{resource} | jq '{relevant_fields}')
echo "BEFORE: $BEFORE_STATE"

# 2. Perform the action
ACTION_RESULT=$(curl -s -X POST ... | jq .)
echo "ACTION RESULT: $ACTION_RESULT"

# 3. Record AFTER state
AFTER_STATE=$(curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8006/api/{resource} | jq '{relevant_fields}')
echo "AFTER: $AFTER_STATE"

# 4. Log the comparison
echo "=== STATE CHANGE VERIFICATION ==="
echo "Before: $BEFORE_STATE"
echo "After: $AFTER_STATE"
echo "Expected change: {describe what should have changed}"

Mock-provider pattern (deterministic, $0)

For timeout/latency/error-handling behavior that would otherwise need a real LLM call, point the OpenAI SDK at a local mock instead — it honours OPENAI_BASE_URL, so a small local Responses-API server can stand in for the provider while everything downstream (executor, credentials, billing) still runs for real. This proved out 4/4 test items at $0 in this run. This covers LLM blocks (providers.py) and the Codex block — the copilot's own LLM calls go through backend/util/clients.py, which passes base_url explicitly and does not read OPENAI_BASE_URL, so this trick doesn't reach copilot chat.

# In $BACKEND_DIR/.env, point the OpenAI provider at a local mock server
# that implements the subset of the Responses API you need (e.g. delayed
# responses to test timeout handling, or a 500 to test error surfacing).
# Restart the backend after changing this — it's read at startup.
OPENAI_BASE_URL=http://localhost:{mock_port}/v1

Use a throwaway/dummy provider credential with the mock, never the system credential — only the transport is faked, so the credential lookup still runs for real and whatever key you configure gets sent as a header to your local mock server. A dummy key also means the mock server's logs (which may end up pasted into a PR comment) can't leak a real one. Aside from the credential, this is safe to use for anything that isn't itself testing model output quality.

In docker mode, localhost inside the backend container isn't reachable from your host-side mock server — point OPENAI_BASE_URL at a Compose-reachable hostname instead (or host.docker.internal with a host-gateway entry), and set it before docker compose up or restart the affected services after changing $BACKEND_DIR/.env.

Keep the mock server's own response timeout short — the OpenAI SDK's default client timeout is 600s, so a hung mock stalls every LLM-backed block for that long instead of failing fast.

Browser testing with agent-browser

Primary tool — use this wherever agent-browser is installed:

# Close any existing session
agent-browser close 2>/dev/null || true

# Use --session-name to persist cookies across navigations
# This means login only needs to happen once per test session
agent-browser --session-name pr-test open 'http://localhost:3000/login' --timeout 15000

# Get interactive elements
agent-browser --session-name pr-test snapshot | grep "textbox\|button"

# Login (read creds from env — set PR_TEST_USER_EMAIL / PR_TEST_USER_PASSWORD or ask the user)
agent-browser --session-name pr-test fill {email_ref} "$PR_TEST_USER_EMAIL"
agent-browser --session-name pr-test fill {password_ref} "$PR_TEST_USER_PASSWORD"
agent-browser --session-name pr-test click {login_button_ref}
sleep 5

# Dismiss cookie banner if present
agent-browser --session-name pr-test click 'text=Accept All' 2>/dev/null || true

# Navigate — cookies are preserved so login persists
agent-browser --session-name pr-test open 'http://localhost:3000/copilot' --timeout 10000

# Take screenshot
agent-browser --session-name pr-test screenshot $RESULTS_DIR/01-page.png

# Interact with elements
agent-browser --session-name pr-test fill {ref} "text"
agent-browser --session-name pr-test press "Enter"
agent-browser --session-name pr-test click {ref}
agent-browser --session-name pr-test click 'text=Button Text'

# Read page content
agent-browser --session-name pr-test snapshot | grep "text:"

Key pages:

• /copilot — CoPilot chat (for testing copilot features)
• /build — Agent builder (for testing block/node features)
• /build?flowID={id} — Specific agent in builder
• /library — Agent library (for testing listing/import features)
• /library/agents/{id} — Agent detail with run history
• /marketplace — Marketplace

Fallback — if agent-browser isn't installed on the host, don't let npx download it. Use @playwright/test instead — the package is already in the frontend's node_modules, but the Chromium binary itself is not; Playwright caches browser binaries separately (~/.cache/ms-playwright/ by default) and nothing installs them automatically. Run pnpm exec playwright install chromium once per host if chromium.launch() fails looking for an executable, then use it for finer control over timing (waitForSelector, explicit timeouts) than agent-browser's CLI gives you:

cd $FRONTEND_DIR  # required — @playwright/test is only declared here, not at the repo root
# Check the installer's own exit status, not grep's — piping through grep to
# drop blank lines would otherwise swallow a real install failure and let
# chromium.launch() below fail later with a much less clear error.
pnpm exec playwright install chromium
[ $? -eq 0 ] || { echo "ERROR: playwright install chromium failed"; exit 1; }

node -e "
const { chromium } = require('@playwright/test');
(async () => {
  const browser = await chromium.launch();
  try {
    const page = await browser.newPage();
    await page.goto('http://localhost:3000/login');
    await page.screenshot({ path: '$RESULTS_DIR/01-login.png' });
  } finally {
    await browser.close();  // otherwise a goto timeout leaks a headless Chromium process
  }
})();
"

Checking logs

Native mode: when running via poetry run app + pnpm dev, all app logs stream to the .ign.*.logs files written by the tee pipes in 3e-native. rest_server, executor, copilot_executor, websocket, scheduler, notification_server, and database_manager are all subprocesses of the single poetry run app parent, so their output is interleaved in .ign.application.logs.

# Backend (all app subprocesses interleaved)
tail -f $BACKEND_DIR/.ign.application.logs

# Frontend (Next.js dev server)
tail -f $FRONTEND_DIR/.ign.frontend.logs

# Filter for errors across either log
grep -iE "error|exception|traceback" $BACKEND_DIR/.ign.application.logs | tail -20
grep -iE "error|exception|traceback" $FRONTEND_DIR/.ign.frontend.logs | tail -20

Docker mode:

# Backend REST server
docker logs autogpt_platform-rest_server-1 2>&1 | tail -30

# Executor (runs agent graphs)
docker logs autogpt_platform-executor-1 2>&1 | tail -30

# Co

…(truncated)