Specs E2e Verification

Specs E2E Verification

Overview

Performs real environment verification after a specification has been implemented and cleaned up. This skill bridges the gap between unit-tested code and observable runtime behavior by:

1. Detecting the application type from project files
2. Starting the local runtime (Docker Compose, dev server, Spring Boot, etc.)
3. Deriving tests from [IMP] acceptance criteria in the functional specification
4. Executing real tests (curl, Playwright, computer-use)
5. Mapping results to acceptance criteria
6. Generating a verification report
7. Tearing down the environment

Input: docs/specs/[id]/ (spec folder with functional specification and tasks)
Output: docs/specs/[id]/e2e-report-YYYY-MM-DD-HHMMSS.md

When to Use

• Use after specs.task-implementation and specs.code-cleanup to confirm the feature works in reality.
• Use when a developer says "test it for real", "verify the API actually works", "run e2e checks", or "validate acceptance criteria live".
• Use to generate evidence of feature completion before closing a specification.
• Do NOT use for unit testing, static analysis, or code review — this is runtime behavioral verification only.

Arguments

Best Practices

• Safety first: Never run destructive commands (rm -rf, docker system prune, sudo).
• Detect, don’t assume: Use file heuristics to determine the app type; ask the user only when ambiguous.
• AC-driven: Every test must trace back to an [IMP] acceptance criterion from the functional specification.
• Clean up: Always teardown unless --keep-alive is passed; warn about leftover processes.
• No code changes: This skill is read-only regarding source code. It may create reports but never patches logic.
• Use TodoWrite: Track progress across all 8 phases.

Instructions

Phase 1: Parse Arguments and Load Context

1. Parse $ARGUMENTS:

   • --spec (required): spec folder path. Validate that the directory exists and contains at least one functional specification file (YYYY-MM-DD--*.md). If missing or invalid, abort with an error.
   • --task (optional): task ID filter (e.g., TASK-003). If provided, validate that tasks/<task-id>.md exists inside the spec folder.
   • --keep-alive (optional): boolean flag. If present, skip teardown at the end.
   • --timeout (optional): positive integer in seconds. Default is 120. Validate that the value is a positive integer; if not, abort with an error.
   • --insecure (optional): boolean flag. If present, curl commands MAY use -k / --insecure for local development with self-signed certificates. By default, TLS bypass is forbidden (REQ-NR003).

2. Read the functional specification and extract:

   • All acceptance criteria with their taxonomy tags ([IMP], [SEF], [EXT])
   • Only [IMP] criteria will generate runtime tests

3. If --task is provided, read the task file and limit scope to its provides files and related AC.

4. Use TodoWrite to create a todo list for all 8 phases.

Phase 1.5: Security Validation Gate

Before any command is executed, run the following security checks:

1. Command Whitelist Check:

   • The startup command derived in Phase 3 MUST match a pattern documented in references/test-execution-patterns.md.
   • IF the command is NOT in the whitelist → use AskUserQuestion to request explicit user confirmation before execution.
   • Whitelisted commands include: docker compose up -d --build, ./mvnw spring-boot:run, ./gradlew bootRun, npm run dev, npm run start:dev, npm start, cargo tauri build --debug, cargo tauri dev, npm run electron:dev, npx electron ., open *.app, and equivalent local process launchers.
   • Any command containing sudo, rm -rf, docker system prune, mkfs, dd, or similar destructive operations is NOT whitelisted and SHALL be rejected.

2. Forbidden Pattern Scan (REQ-NR001):

   • Scan the derived startup command and all generated test commands for:
     • sudo → abort with: "Forbidden: sudo is not permitted during E2E verification."
     • rm -rf → abort with: "Forbidden: rm -rf is not permitted during E2E verification."
     • docker system prune → abort with: "Forbidden: docker system prune is not permitted during E2E verification."
     • Any rm, drop, destroy, prune targeting databases, volumes, or local data → abort with: "Forbidden: destructive data operations are not permitted."
   • IF any forbidden pattern is detected → abort immediately; do NOT proceed to startup.

3. TLS Enforcement Check (REQ-NR003):

   • For any generated curl command:
     • IF it contains -k or --insecure AND --insecure was NOT passed → abort with: "Forbidden: curl TLS bypass (-k / --insecure) is disabled by default. Pass --insecure to opt-in."
     • IF --insecure was passed → log a warning: "WARNING: TLS certificate verification is disabled. Use only for local development."

4. Data Integrity Pre-Check (REQ-NR004):

   • Before executing startup commands, inspect them for patterns that would overwrite or delete existing data (e.g., rm, drop, prune, volume deletion flags).
   • IF the command would modify existing databases, volumes, or local data directories → abort with: "Forbidden: startup commands must not overwrite or delete existing data."

Phase 2: Detect Application Type and Discover Port

1. Set PROJECT_ROOT to the directory containing .git or the parent directory of --spec.

2. Inspect PROJECT_ROOT for configuration files using the following heuristics (execute in order):

   Docker-managed (highest priority):

   [ -f "$PROJECT_ROOT/docker-compose.yml" ] || [ -f "$PROJECT_ROOT/docker-compose.yaml" ] || [ -f "$PROJECT_ROOT/compose.yml" ]
   

   If any of these files exist, classify as Docker-managed regardless of other framework configs.

   JVM / Spring Boot:

   [ -f "$PROJECT_ROOT/pom.xml" ] || [ -f "$PROJECT_ROOT/build.gradle" ] || [ -f "$PROJECT_ROOT/build.gradle.kts" ]
   

   AND verify source directory exists:

   [ -d "$PROJECT_ROOT/src/main/java" ]
   

   If both conditions are true, classify as JVM-based service.

   NestJS:

   [ -f "$PROJECT_ROOT/package.json" ] && grep -q '"@nestjs/core"' "$PROJECT_ROOT/package.json"
   

   If true, classify as NestJS.

   Web SPA (React / Vue / Angular):

   [ -f "$PROJECT_ROOT/package.json" ] && ( grep -q '"react"' "$PROJECT_ROOT/package.json" || grep -q '"vue"' "$PROJECT_ROOT/package.json" || grep -q '"@angular/core"' "$PROJECT_ROOT/package.json" )
   

   If true, classify as Web SPA.

   Desktop App:

   [ -f "$PROJECT_ROOT/src-tauri/Cargo.toml" ] || ( [ -f "$PROJECT_ROOT/package.json" ] && grep -q '"electron"' "$PROJECT_ROOT/package.json" ) || [ -n "$(find "$PROJECT_ROOT" -maxdepth 2 -name '*.csproj' -print -quit 2>/dev/null)" ]
   

   If true, classify as Desktop App.

   Python:

   [ -f "$PROJECT_ROOT/requirements.txt" ] || [ -f "$PROJECT_ROOT/pyproject.toml" ] || [ -f "$PROJECT_ROOT/app.py" ] || [ -f "$PROJECT_ROOT/manage.py" ]
   

   If true, classify as Python.

3. Apply priority rules:

   • Docker Compose is always prioritized: If a Docker Compose file exists AND any framework config also exists, classify as Docker-managed. The compose stack defines the runtime.
   • If multiple non-Docker configs are detected (e.g., both pom.xml and package.json without Docker Compose):
     • If the spec's domain clearly indicates backend vs frontend (e.g., spec title contains "API", "backend", "service"), prefer JVM or NestJS.
     • If the spec's domain clearly indicates frontend (e.g., spec title contains "UI", "page", "component"), prefer Web SPA.
     • If still ambiguous, proceed to user prompt (step 4).

4. If no recognizable config is found, OR if multiple non-Docker configs exist and the spec domain is ambiguous, use AskUserQuestion with exactly these options:

   • "REST API"
   • "Web SPA"
   • "Desktop"
   • "Skip"

5. Port Discovery: Once the application type is known, determine the target port by inspecting framework configuration files in this order:

   • Vite projects (vite.config.ts or vite.config.js):

     grep -oE 'port:\s*[0-9]+' "$PROJECT_ROOT/vite.config.ts" 2>/dev/null | grep -oE '[0-9]+' || \
     grep -oE 'port:\s*[0-9]+' "$PROJECT_ROOT/vite.config.js" 2>/dev/null | grep -oE '[0-9]+'
     

   • Spring Boot (application.yml):

     grep -A5 '^server:' "$PROJECT_ROOT/src/main/resources/application.yml" 2>/dev/null | grep 'port:' | head -1 | tr -dc '0-9'
     

   • Spring Boot (application.properties):

     grep '^server.port=' "$PROJECT_ROOT/src/main/resources/application.properties" 2>/dev/null | cut -d= -f2 | tr -dc '0-9'
     

   • Node.js / package.json scripts:

     grep -oE -- '--port [0-9]+' "$PROJECT_ROOT/package.json" 2>/dev/null | grep -oE '[0-9]+' | head -1
     

     Also check for PORT environment variable in scripts:

     grep -oE 'PORT=[0-9]+' "$PROJECT_ROOT/package.json" 2>/dev/null | grep -oE '[0-9]+' | head -1
     

   • Fallback defaults (if no port is found in any config file):

     App Type	Default Port
     Node.js / NestJS	3000
     Spring Boot (JVM)	8080
     Angular	4200
     Vite (React/Vue)	5173
     Python	8000

6. Log the detected type and discovered port; both will be recorded in the report.

Phase 3: Start Environment and Wait for Readiness

Read references/test-execution-patterns.md (shipped with this skill) for the command mapping. Based on detection:

Data integrity pre-flight (REQ-NR004): Before executing the startup command, verify it does not contain patterns that overwrite or delete existing databases, volumes, or local data (e.g., rm, --volumes, prune, drop). If a destructive pattern is detected, abort immediately with: "Startup aborted: command would destroy existing data."

1. Initialize runtime state:

   • STARTUP_COMMAND="" — the exact command used to start the environment
   • HEALTH_CHECK_METHOD="" — description of how readiness was determined
   • STARTUP_LOGS_FILE="$(mktemp)" — temp file capturing stdout/stderr from startup
   • STARTUP_TIMEOUT="${TIMEOUT:-120}" — seconds to wait for readiness
   • START_TIME="$(date +%s)"
   • STARTUP_PID="" — background process PID (for local processes)

2. Pre-startup port check (all types):

   if lsof -i :"$TARGET_PORT" >/dev/null 2>&1 || nc -z localhost "$TARGET_PORT" 2>/dev/null; then
       echo "Port $TARGET_PORT is already in use."
       EXISTING_PID=$(lsof -ti:"$TARGET_PORT" | head -n1)
       EXISTING_CMD=$(ps -p "$EXISTING_PID" -o comm= 2>/dev/null || echo "unknown")
       if [ "$APP_TYPE" = "Docker-managed" ] && docker ps --format '{{.Names}}' 2>/dev/null | grep -q "$EXISTING_CMD"; then
           echo "Existing Docker container detected on port $TARGET_PORT; reusing it."
           STARTUP_COMMAND="(existing container reused)"
       elif [ "$APP_TYPE" = "JVM-based service" ] && echo "$EXISTING_CMD" | grep -q "java"; then
           echo "Existing Java process detected on port $TARGET_PORT; reusing it."
           STARTUP_COMMAND="(existing Java process reused)"
       else
           echo "Port $TARGET_PORT is occupied by an unrelated process ($EXISTING_CMD). Free the port and retry."
           exit 1
       fi
   fi
   

3. Docker-managed startup:

   • Verify Docker daemon is reachable:

     if ! docker info >/dev/null 2>&1; then
         echo "Docker is not available. Please start Docker Desktop or use direct framework startup."
         exit 1
     fi
     

   • Set STARTUP_COMMAND="docker compose up -d --build"
   • Execute:

     cd "$PROJECT_ROOT" && docker compose up -d --build >> "$STARTUP_LOGS_FILE" 2>&1
     

   • Health check loop (timeout enforced):

     HEALTH_CHECK_METHOD="docker compose ps --format json"
     READY=false
     while [ $(( $(date +%s) - START_TIME )) -lt "$STARTUP_TIMEOUT" ]; do
         # Option A: Docker native health status (supports both JSON array and NDJSON)
         if docker compose ps --format json 2>/dev/null | jq -s -e '.[] | select(.Health=="healthy")' >/dev/null 2>&1; then
             READY=true
             HEALTH_CHECK_METHOD="docker compose ps (HEALTHCHECK=healthy)"
             break
         fi
         # Option B: Fallback port polling if no HEALTHCHECK defined
         if curl -sf "http://localhost:${TARGET_PORT}" >/dev/null 2>&1; then
             READY=true
             HEALTH_CHECK_METHOD="port polling via curl on localhost:${TARGET_PORT} (every 3s)"
             break
         fi
         sleep 3
     done
     if [ "$READY" != "true" ]; then
         echo "Startup timeout (${STARTUP_TIMEOUT}s) exceeded for Docker Compose."
         docker compose logs --tail=50 >> "$STARTUP_LOGS_FILE" 2>&1
         # Attempt cleanup to avoid orphan containers (REQ-NR006)
         docker compose down >/dev/null 2>&1 || true
         exit 1
     fi
     

4. Spring Boot startup:

   • Determine build tool and launch in background with log capture:

     if [ -f "$PROJECT_ROOT/pom.xml" ]; then
         cd "$PROJECT_ROOT" && ./mvnw spring-boot:run -Dspring-boot.run.profiles=e2e >> "$STARTUP_LOGS_FILE" 2>&1 &
         STARTUP_PID=$!
         STARTUP_COMMAND="./mvnw spring-boot:run -Dspring-boot.run.profiles=e2e"
     elif [ -f "$PROJECT_ROOT/build.gradle" ] || [ -f "$PROJECT_ROOT/build.gradle.kts" ]; then
         cd "$PROJECT_ROOT" && ./gradlew bootRun --args='--spring.profiles.active=e2e' >> "$STARTUP_LOGS_FILE" 2>&1 &
         STARTUP_PID=$!
         STARTUP_COMMAND="./gradlew bootRun --args='--spring.profiles.active=e2e'"
     fi
     

   • Health check loop (timeout enforced):

     HEALTH_CHECK_METHOD="Spring Boot actuator /actuator/health"
     READY=false
     while [ $(( $(date +%s) - START_TIME )) -lt "$STARTUP_TIMEOUT" ]; do
         # Option A: Actuator health endpoint
         if curl -sf "http://localhost:${TARGET_PORT}/actuator/health" >/dev/null 2>&1; then
             READY=true
             HEALTH_CHECK_METHOD="Spring Boot actuator /actuator/health"
             break
         fi
         # Option B: Fallback to raw port readiness
         if nc -z localhost "$TARGET_PORT" 2>/dev/null; then
             READY=true
             HEALTH_CHECK_METHOD="port polling via nc on localhost:${TARGET_PORT} (every 3s)"
             break
         fi
         # Fail fast if the background process exited early
         if [ -n "$STARTUP_PID" ] && ! kill -0 "$STARTUP_PID" 2>/dev/null; then
             echo "Spring Boot process exited before reaching healthy state."
             break
         fi
         sleep 3
     done
     if [ "$READY" != "true" ]; then
         echo "Startup timeout (${STARTUP_TIMEOUT}s) exceeded for Spring Boot."
         # Capture last lines of startup logs for the report
         tail -n 100 "$STARTUP_LOGS_FILE" >> "$STARTUP_LOGS_FILE".final 2>&1 || true
         # Attempt cleanup (REQ-NR006)
         [ -n "$STARTUP_PID" ] && kill -TERM "$STARTUP_PID" 2>/dev/null || true
         exit 1
     fi
     

5. NestJS / Node.js startup:

   • Verify node_modules exists to avoid cryptic errors:

     if [ ! -d "$PROJECT_ROOT/node_modules" ]; then
         echo "node_modules not found. Run 'npm install' before verification."
         exit 1
     fi
     

   • Determine the startup command from package.json scripts:

     if [ "$APP_TYPE" = "NestJS" ] && grep -q '"start:dev"' "$PROJECT_ROOT/package.json" 2>/dev/null; then
         cd "$PROJECT_ROOT" && npm run start:dev >> "$STARTUP_LOGS_FILE" 2>&1 &
         STARTUP_PID=$!
         STARTUP_COMMAND="npm run start:dev"
     elif grep -q '"dev"' "$PROJECT_ROOT/package.json" 2>/dev/null; then
         cd "$PROJECT_ROOT" && npm run dev >> "$STARTUP_LOGS_FILE" 2>&1 &
         STARTUP_PID=$!
         STARTUP_COMMAND="npm run dev"
     else
         cd "$PROJECT_ROOT" && npm start >> "$STARTUP_LOGS_FILE" 2>&1 &
         STARTUP_PID=$!
         STARTUP_COMMAND="npm start"
     fi
     

   • Health check loop (timeout enforced):

     HEALTH_CHECK_METHOD="port polling via nc/curl on localhost:${TARGET_PORT} (every 3s)"
     READY=false
     while [ $(( $(date +%s) - START_TIME )) -lt "$STARTUP_TIMEOUT" ]; do
         # Option A: HTTP readiness via curl
         if curl -sf "http://localhost:${TARGET_PORT}" >/dev/null 2>&1; then
             READY=true
             HEALTH_CHECK_METHOD="curl on localhost:${TARGET_PORT}"
             break
         fi
         # Option B: Raw port readiness via nc
         if nc -z localhost "$TARGET_PORT" 2>/dev/null; then
             READY=true
             HEALTH_CHECK_METHOD="nc -z localhost:${TARGET_PORT}"
             break
         fi
         # Fail fast if the background process exited early
         if [ -n "$STARTUP_PID" ] && ! kill -0 "$STARTUP_PID" 2>/dev/null; then
             echo "Node.js/NestJS process exited before reaching healthy state."
             break
         fi
         sleep 3
     done
     if [ "$READY" != "true" ]; then
         echo "Startup timeout (${STARTUP_TIMEOUT}s) exceeded for Node.js/NestJS."
         tail -n 100 "$STARTUP_LOGS_FILE" >> "$STARTUP_LOGS_FILE".final 2>&1 || true
         # Attempt cleanup (REQ-NR006)
         [ -n "$STARTUP_PID" ] && kill -TERM "$STARTUP_PID" 2>/dev/null || true
         exit 1
     fi
     

   • Console ready detection: After readiness is confirmed, scan $STARTUP_LOGS_FILE for common server-ready messages (e.g., Nest application successfully started, Local:, ready in, Server running) and record the first matching line in the report as evidence of successful startup.

6. Python (FastAPI / Django / Flask) (see references/test-execution-patterns.md):

   • Run uvicorn main:app --reload, python manage.py runserver, or flask run
   • Wait for port readiness

7. Desktop App (Tauri / Electron / .NET MAUI) (see references/test-execution-patterns.md):

   Framework detection:

   DESKTOP_FRAMEWORK=""
   if [ -f "$PROJECT_ROOT/src-tauri/Cargo.toml" ]; then
       DESKTOP_FRAMEWORK="tauri"
   elif [ -f "$PROJECT_ROOT/package.json" ] && grep -q '"electron"' "$PROJECT_ROOT/package.json"; then
       DESKTOP_FRAMEWORK="electron"
   elif [ -n "$(find "$PROJECT_ROOT" -maxdepth 2 -name '*.csproj' -print -quit 2>/dev/null)" ]; then
       DESKTOP_FRAMEWORK="dotnet-maui"
   fi
   

   Build step (triggered when no pre-built debug binary exists or when source is newer than target):

   • Tauri:

     if [ "$DESKTOP_FRAMEWORK" = "tauri" ]; then
         if ! command -v cargo >/dev/null 2>&1; then
             echo "ERROR: Rust/Cargo is not installed. Tauri build requires cargo."
             exit 1
         fi
         BUILD_COMMAND="cargo tauri build --debug"
         cd "$PROJECT_ROOT" && $BUILD_COMMAND >> "$STARTUP_LOGS_FILE" 2>&1
         BUILD_EXIT_CODE=$?
         if [ "$BUILD_EXIT_CODE" -ne 0 ]; then
             echo "Tauri build failed (exit code $BUILD_EXIT_CODE). See startup logs for details."
             exit 1
         fi
         STARTUP_COMMAND="$BUILD_COMMAND (build succeeded)"
     fi
     

   • Electron:

     if [ "$DESKTOP_FRAMEWORK" = "electron" ]; then
         if [ ! -d "$PROJECT_ROOT/node_modules" ]; then
             echo "ERROR: node_modules not found. Run 'npm install' before verification."
             exit 1
         fi
         BUILD_COMMAND="(no separate build required for dev mode)"
         STARTUP_COMMAND="npm run electron:dev"
         if grep -q '"electron:build"' "$PROJECT_ROOT/package.json" 2>/dev/null; then
             BUILD_COMMAND="npm run electron:build"
             cd "$PROJECT_ROOT" && $BUILD_COMMAND >> "$STARTUP_LOGS_FILE" 2>&1
             BUILD_EXIT_CODE=$?
             if [ "$BUILD_EXIT_CODE" -ne 0 ]; then
                 echo "Electron build failed (exit code $BUILD_EXIT_CODE). See startup logs for details."
                 exit 1
             fi
         fi
     fi
     

   Launch the built application binary:

   • Tauri (macOS):

     APP_BUNDLE=$(find "$PROJECT_ROOT/src-tauri/target/debug/bundle" -name "*.app" -print -quit 2>/dev/null)
     if [ -n "$APP_BUNDLE" ]; then
         open "$APP_BUNDLE" >> "$STARTUP_LOGS_FILE" 2>&1 &
         STARTUP_PID=$!
         STARTUP_COMMAND="open $APP_BUNDLE"
     else
         DEV_BINARY=$(find "$PROJECT_ROOT/src-tauri/target/debug" -maxdepth 1 -type f -executable ! -name '*.dylib' ! -name '*.so' -print -quit 2>/dev/null)
         if [ -n "$DEV_BINARY" ]; then
             "$DEV_BINARY" >> "$STARTUP_LOGS_FILE" 2>&1 &
             STARTUP_PID=$!
             STARTUP_COMMAND="$DEV_BINARY"
         else
             cargo tauri dev >> "$STARTUP_LOGS_FILE" 2>&1 &
             STARTUP_PID=$!
             STARTUP_COMMAND="cargo tauri dev"
         fi
     fi
     

   • Tauri (Linux):

     APP_BINARY=$(find "$PROJECT_ROOT/src-tauri/target/debug" -maxdepth 1 -type f -executable ! -name '*.so' -print -quit 2>/dev/null)
     if [ -n "$APP_BINARY" ]; then
         "$APP_BINARY" >> "$STARTUP_LOGS_FILE" 2>&1 &
         STARTUP_PID=$!
         STARTUP_COMMAND="$APP_BINARY"
     else
         cargo tauri dev >> "$STARTUP_LOGS_FILE" 2>&1 &
         STARTUP_PID=$!
         STARTUP_COMMAND="cargo tauri dev"
     fi
     

   • Electron:

     if grep -q '"electron:dev"' "$PROJECT_ROOT/package.json" 2>/dev/null; then
         cd "$PROJECT_ROOT" && npm run electron:dev >> "$STARTUP_LOGS_FILE" 2>&1 &
         STARTUP_PID=$!
         STARTUP_COMMAND="npm run electron:dev"
     elif grep -q '"start"' "$PROJECT_ROOT/package.json" 2>/dev/null; then
         cd "$PROJECT_ROOT" && npm start >> "$STARTUP_LOGS_FILE" 2>&1 &
         STARTUP_PID=$!
         STARTUP_COMMAND="npm start"
     else
         cd "$PROJECT_ROOT" && npx electron . >> "$STARTUP_LOGS_FILE" 2>&1 &
         STARTUP_PID=$!
         STARTUP_COMMAND="npx electron ."
     fi
     

   Health check (process appearance, timeout enforced):

   HEALTH_CHECK_METHOD="process polling via ps/kill -0 (every 3s)"
   READY=false
   while [ $(( $(date +%s) - START_TIME )) -lt "$STARTUP_TIMEOUT" ]; do
       if [ -n "$STARTUP_PID" ] && kill -0 "$STARTUP_PID" 2>/dev/null; then
           READY=true
           HEALTH_CHECK_METHOD="process PID ${STARTUP_PID} confirmed alive"
           break
       fi
       sleep 3
   done
   if [ "$READY" != "true" ]; then
       echo "Startup timeout (${STARTUP_TIMEOUT}s) exceeded for Desktop app."
       tail -n 100 "$STARTUP_LOGS_FILE" >> "$STARTUP_LOGS_FILE".final 2>&1 || true
       [ -n "$STARTUP_PID" ] && kill -TERM "$STARTUP_PID" 2>/dev/null || true
       exit 1
   fi
   

8. Post-startup bookkeeping (all types):

   • Record STARTUP_COMMAND and HEALTH_CHECK_METHOD in report metadata (AC-010).
   • Append STARTUP_LOGS_FILE contents to the report under Raw Output (REQ-020).
   • If startup fails for any reason, capture all available logs and abort with a clear error message.

Phase 4: Generate and Execute Tests

CRITICAL: Only test [IMP] acceptance criteria. Translate each into one or more concrete runtime actions.

REST API Tests (curl)

Prerequisite check: Before generating any tests, verify curl is installed:

if ! command -v curl >/dev/null 2>&1; then
    echo "ERROR: curl is not installed."
    echo "Install instructions:"
    echo "  macOS:    brew install curl"
    echo "  Ubuntu:   sudo apt-get install curl"
    echo "  Windows:  choco install curl   or   winget install curl"
    exit 1
fi
if ! command -v jq >/dev/null 2>&1; then
    echo "WARNING: jq is not installed. JSON body assertions will fall back to grep (less precise)."
    echo "Install instructions:"
    echo "  macOS:    brew install jq"
    echo "  Ubuntu:   sudo apt-get install jq"
    echo "  Windows:  choco install jq   or   winget install jqlang.jq"
fi

1. Parse [IMP] AC for endpoint hints

For each [IMP] acceptance criterion in the specification:

• Extract the HTTP method by searching for keywords: GET, POST, PUT, PATCH, DELETE (case-insensitive).
• Extract the endpoint path by searching for patterns starting with / followed by alphanumeric segments, e.g., /api/users, /v1/health.
• Extract the expected HTTP status code by searching for numeric patterns 2xx, 3xx, 4xx, 5xx or specific codes like 200, 201, 204, 400, 401, 403, 404, 500.
• Extract expected Content-Type by searching for application/json, text/plain, text/html, etc.
• Extract expected response body hints (field names, array presence, string values) from the AC text.

If an [IMP] AC does not contain a parseable endpoint path and method, mark it MANUAL CHECK REQUIRED and skip to the next criterion.

2. Discover authentication credentials

Before constructing curl commands, attempt to locate test credentials by scanning the following files in PROJECT_ROOT (in order):

Discovery logic:

# .env files
[ -f "$PROJECT_ROOT/.env.test" ] && export $(grep -E '^(E2E_AUTH_TOKEN|E2E_USERNAME|E2E_PASSWORD)=' "$PROJECT_ROOT/.env.test" | xargs)
[ -f "$PROJECT_ROOT/.env.local" ] && export $(grep -E '^(E2E_AUTH_TOKEN|E2E_USERNAME|E2E_PASSWORD)=' "$PROJECT_ROOT/.env.local" | xargs)

# Spring YAML
[ -f "$PROJECT_ROOT/src/main/resources/application-test.yml" ] && \
  E2E_AUTH_TOKEN=$(grep -A1 'e2e:' "$PROJECT_ROOT/src/main/resources/application-test.yml" | grep 'auth-token:' | sed 's/.*: *//')

# Spring properties
[ -f "$PROJECT_ROOT/src/main/resources/application-test.properties" ] && \
  E2E_AUTH_TOKEN=$(grep '^e2e.auth-token=' "$PROJECT_ROOT/src/main/resources/application-test.properties" | cut -d= -f2-)

# JSON credentials file
[ -f "$PROJECT_ROOT/e2e.credentials.json" ] && \
  E2E_AUTH_TOKEN=$(jq -r '.E2E_AUTH_TOKEN // empty' "$PROJECT_ROOT/e2e.credentials.json")

If no credentials are found after scanning all files AND the AC text implies authentication is required (mentions "auth", "login", "token", "protected", "bearer", "API key"), use AskUserQuestion to prompt the user:

• "Enter E2E_AUTH_TOKEN (or leave blank if none)"
• "Enter E2E_USERNAME (or leave blank if none)"
• "Enter E2E_PASSWORD (or leave blank if none)"

Security: Redact token values in the E2E report; show only the header name (e.g., Authorization: Bearer <redacted>).

3. Generate curl commands

For each parseable [IMP] AC, construct the curl command using this exact pattern:

curl -s -w "\n%{http_code}" -o /tmp/e2e_resp.json \
  -X <METHOD> \
  -H "Content-Type: application/json" \
  <AUTH_HEADER> \
  -d '<REQUEST_BODY>' \
  "http://localhost:${TARGET_PORT}<PATH>"

Rules:

• Always include -s -w "\n%{http_code}" -o /tmp/e2e_resp.json.
• -X <METHOD>: only add if the method is not GET. For GET, omit -X entirely.
• -H "Content-Type: application/json": only add for POST, PUT, PATCH.
• -d '<REQUEST_BODY>': only add when the AC describes a request body. If no body is described, omit -d.
• <AUTH_HEADER>:
  • If E2E_AUTH_TOKEN is set: -H "Authorization: Bearer ${E2E_AUTH_TOKEN}"
  • If E2E_USERNAME and E2E_PASSWORD are set: -u "${E2E_USERNAME}:${E2E_PASSWORD}"
  • If no credentials found: omit auth header.
• TLS enforcement (REQ-NR003): NEVER add -k or --insecure to curl commands unless the --insecure flag was explicitly passed when invoking the skill. If --insecure was passed, log a warning that TLS verification is disabled.
• Store the raw response body in /tmp/e2e_resp.json and the status code on the last line of stdout.

4. Execute curl and assert (no retry)

Execute each curl command exactly once (REQ-NR008). Do NOT retry on failure.

HTTP_CODE=$(curl -s -w "\n%{http_code}" -o /tmp/e2e_resp.json <curl args> | tail -n 1)

Assertions (all must pass for the AC to be VERIFIED):

a. HTTP status code:

if [ "$HTTP_CODE" -ne "$EXPECTED_STATUS" ]; then
    echo "FAIL: Expected status $EXPECTED_STATUS, got $HTTP_CODE"
    STATUS="FAILED"
fi

b. Content-Type header (only if specified in the AC):

ACTUAL_CT=$(curl -s -o /dev/null -D - <curl args> | grep -i "Content-Type:" | head -1 | sed 's/Content-Type: //i' | tr -d '\r')
if [ -n "$EXPECTED_CT" ] && ! echo "$ACTUAL_CT" | grep -qi "$EXPECTED_CT"; then
    echo "FAIL: Expected Content-Type '$EXPECTED_CT', got '$ACTUAL_CT'"
    STATUS="FAILED"
fi

c. Response body structure using jq (preferred) or grep (fallback):

• If jq is installed and the response is JSON:

  # Assert field exists
  jq -e '.fieldName' /tmp/e2e_resp.json >/dev/null || { echo "FAIL: Missing .fieldName"; STATUS="FAILED"; }
  
  # Assert field equals expected value
  jq -e '.fieldName == "expectedValue"' /tmp/e2e_resp.json >/dev/null || { echo "FAIL: .fieldName mismatch"; STATUS="FAILED"; }
  
  # Assert array length
  jq -e '(.items | length) > 0' /tmp/e2e_resp.json >/dev/null || { echo "FAIL: .items is empty"; STATUS="FAILED"; }
  
  # Assert nested field
  jq -e '.data.user.email' /tmp/e2e_resp.json >/dev/null || { echo "FAIL: Missing .data.user.email"; STATUS="FAILED"; }
  

• If jq is NOT installed, use grep as fallback:

  grep -q '"fieldName"' /tmp/e2e_resp.json || { echo "FAIL: Missing fieldName"; STATUS="FAILED"; }
  

5. Record results

For each curl test, record:

• AC ID and text (truncated)
• Generated curl command (with auth tokens redacted)
• Expected status code, Content-Type, body assertions
• Actual status code, Content-Type, body snippet (first 500 chars)
• Pass/fail status
• Execution time (optional, for report reference)

If any assertion fails, mark the AC as FAILED immediately. Do not retry.

Web SPA Tests (Playwright)

Prerequisite check: Before generating any SPA tests, verify Playwright is installed:

PLAYWRIGHT_VERSION=$(npx playwright --version 2>/dev/null || echo "")
if [ -z "$PLAYWRIGHT_VERSION" ]; then
    echo "ERROR: Playwright is not installed."
    echo "Install instructions:"
    echo "  npm install -D @playwright/test"
    echo "  npx playwright install chromium"
    echo ""
    echo "Skipping all Web SPA tests. Acceptance criteria for SPA interactions will be marked MANUAL CHECK REQUIRED."
    for ac_id in $(get_spa_ac_ids); do
        record_result "$ac_id" "MANUAL CHECK REQUIRED" "Playwright not installed"
    done
    # Continue to next test category; do not abort the whole verification
fi
echo "Playwright version: $PLAYWRIGHT_VERSION"

If Playwright is missing, the skill MUST report the gap with the install commands above and skip SPA tests. Do NOT attempt to auto-install.

1. Prepare artifact directory

ARTIFACT_DIR="${SPEC_FOLDER}/e2e-artifacts"
mkdir -p "$ARTIFACT_DIR"

2. Parse [IMP] AC for UI behavior hints

For each [IMP] acceptance criterion in the specification that relates to Web SPA behavior:

• Extract UI interaction keywords: click, fill, type, select, submit, navigate, scroll, hover.
• Extract target selectors by searching for patterns:
  • data-testid="..." or data-testid='...' → [data-testid=...]
  • id="..." or id='...' → #...
  • class="..." or class='...' → .class-name (replace spaces with dots)
  • Button/link text mentions → text=...
  • URL path mentions → /path
• Extract expected visible states: text content expectations, URL expectations, element presence/absence, count expectations.
• Extract form field names and expected input values.

If an [IMP] AC does not contain parseable UI behavior or visible state hints, mark it MANUAL CHECK REQUIRED and skip to the next criterion.

3. Launch headless browser context

Browser MUST be headless by default. Only use headed mode if the user explicitly passes --headed.

For each SPA test, generate a temporary Playwright script and execute it with node:

DEV_SERVER_URL="http://localhost:${TARGET_PORT}"
TEST_SCRIPT="$(mktemp /tmp/e2e-spa-XXXXXX.js)"

cat > "$TEST_SCRIPT" << 'PLAYWRIGHT_EOF'
const { chromium } = require('playwright');
(async () => {
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext({
    viewport: { width: 1280, height: 720 },
    userAgent: 'DeveloperKit-E2E/1.0'
  });
  const page = await context.newPage();
  // Actions and assertions injected here
  await browser.close();
})();
PLAYWRIGHT_EOF

node "$TEST_SCRIPT"

4. Translate AC into Playwright actions

For each parsed UI interaction, generate the corresponding Playwright action inside the temporary script:

Selector precedence (most specific to least specific):

1. [data-testid=...] — preferred, most stable
2. #id — unique element ID
3. .class-name — CSS class
4. [name=...] — form element name
5. text=... — visible text content (fallback)

5. Assert visible states

For each expected visible state, generate the corresponding assertion inside the temporary script:

6. Execute test with timeout enforcement

Each Playwright test MUST enforce a per-test timeout to prevent indefinite hangs (REQ-NR007). The wrapper script uses the timeout command:

TEST_TIMEOUT_SEC=30

npx playwright --version >/dev/null 2>&1 || {
    echo "Playwright not installed; skipping SPA tests."
    record_spa_manual_check
    continue
}

# Build the inline test script
TEST_SCRIPT="$(mktemp /tmp/e2e-spa-XXXXXX.js)"
cat > "$TEST_SCRIPT" << EOF
const { chromium } = require('playwright');
(async () => {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();
  page.setDefaultTimeout(${TEST_TIMEOUT_SEC}000);
  page.setDefaultNavigationTimeout(${TEST_TIMEOUT_SEC}000);
  try {
    await page.goto('${DEV_SERVER_URL}');
    // --- GENERATED ACTIONS ---
    // --- GENERATED ASSERTIONS ---
    console.log('RESULT: PASS');
  } catch (error) {
    console.error('RESULT: FAIL:', error.message);
    const screenshotPath = '${ARTIFACT_DIR}/screenshot-' + Date.now() + '-ac-${AC_ID}.png';
    await page.screenshot({ path: screenshotPath, fullPage: true }).catch(() => {});
    console.error('SCREENSHOT:', screenshotPath);
    process.exitCode = 1;
  } finally {
    await browser.close();
  }
})();
EOF

# Run with timeout wrapper; if the test hangs, it is killed and marked FAILED
if timeout --signal=TERM $((TEST_TIMEOUT_SEC + 5)) node "$TEST_SCRIPT"; then
    STATUS="VERIFIED"
else
    EXIT_CODE=$?
    if [ "$EXIT_CODE" -eq 124 ]; then
        echo "FAIL: Test hung and was terminated after ${TEST_TIMEOUT_SEC}s timeout"
    fi
    STATUS="FAILED"
fi
rm -f "$TEST_SCRIPT"

If the test process hangs beyond the timeout, the timeout command sends SIGTERM, the AC is marked FAILED, and the evidence records: "Test hung and was terminated after ${TEST_TIMEOUT_SEC}s".

7. Screenshot capture on failure

When any assertion or action fails:

1. Capture a full-page screenshot using page.screenshot({ path: ..., fullPage: true }).
2. Save to ${ARTIFACT_DIR}/screenshot-<timestamp>-ac-<AC_ID>.png.
3. Record the screenshot path in the test result evidence.

When a test passes, screenshots are optional and only captured if --capture-success is passed.

8. Record results

For each SPA test, record:

• AC ID and text (truncated)
• Playwright actions executed
• Assertions performed
• Pass/fail status
• Screenshot path (on failure)
• Execution time
• Error message (on failure)

If any assertion fails, mark the AC as FAILED immediately. Do not retry.

Edge cases — error handling:

• Playwright not installed: Report clear error with npm install -D @playwright/test and npx playwright install chromium suggestion. Mark all SPA ACs as MANUAL CHECK REQUIRED. Continue with other test categories.
• Dev server not running: If page.goto() throws net::ERR_CONNECTION_REFUSED or similar, report: "Dev server not reachable at ${DEV_SERVER_URL}. Ensure the server is running before verification." Mark affected ACs as FAILED.
• Browser launch failure: If Chromium fails to launch (e.g., missing system dependencies), report the error and suggest npx playwright install-deps chromium. Mark affected ACs as MANUAL CHECK REQUIRED.

Desktop Tests (Computer-use / MCP)

Prerequisite check: Before generating any desktop tests, verify that computer-use or MCP-based GUI automation tools are available. This skill does NOT auto-install these tools.

GUI_TOOLS_AVAILABLE=false
GUI_TOOL_NAME=""

if [ -n "$CLAUDE_COMPUTER_USE_AVAILABLE" ] || command -v computer-use >/dev/null 2>&1; then
    GUI_TOOLS_AVAILABLE=true
    GUI_TOOL_NAME="computer-use"
fi

if [ -n "$MCP_GUI_SERVER_URL" ] || command -v mcp-gui >/dev/null 2>&1; then
    GUI_TOOLS_AVAILABLE=true
    GUI_TOOL_NAME="mcp-gui"
fi

if [ "$GUI_TOOLS_AVAILABLE" != "tr

…(truncated)