Acceptance Test Runner

Acceptance Test Runner

Execute acceptance test scripts and generate evidence-based validation reports that prove whether implementations meet PRD requirements.

Purpose

Run acceptance tests generated by acceptance-test-writer and produce:

• Binary PASS/FAIL for each criterion
• Evidence (screenshots, API responses) for each test
• Gap analysis for failures with root cause hypotheses
• Actionable feedback for workers to fix issues

When to Use

This skill is invoked:

• By validation-test-agent when --mode=e2e is specified
• Directly when validating against a PRD
• After worker reports "done" to verify implementation

Input Parameters

Example invocations:

# Run all acceptance tests for a PRD
Skill("acceptance-test-runner", args="--prd=PRD-AUTH-001")

# Run specific criterion only
Skill("acceptance-test-runner", args="--prd=PRD-AUTH-001 --criterion=AC-user-login")

# With task traceability
Skill("acceptance-test-runner", args="--prd=PRD-AUTH-001 --task_id=TASK-123")

Output Structure

Reports are stored within the same acceptance-tests/ directory as the test definitions:

acceptance-tests/PRD-XXX/
├── manifest.yaml                  # Test suite metadata
├── AC-user-login.yaml            # Test definitions
├── AC-password-reset.yaml
└── runs/                          # Execution results
    ├── 2026-01-24T10-30-00Z.md    # Timestamped report
    └── evidence/
        ├── AC-user-login-success.png
        ├── AC-user-login-step2.png
        └── AC-api-auth-response.json

Execution Workflow

Step 1: Load Test Definitions

# Verify acceptance tests exist
ls acceptance-tests/${PRD_ID}/

# Load manifest
cat acceptance-tests/${PRD_ID}/manifest.yaml

If no acceptance tests found, return error:

ERROR: No acceptance tests found for ${PRD_ID}
Expected: acceptance-tests/${PRD_ID}/manifest.yaml
Action: Run acceptance-test-writer to generate tests first

Step 2: Prepare Evidence Directory

# Create timestamped evidence directory
TIMESTAMP=$(date -u +"%Y-%m-%dT%H-%M-%SZ")
mkdir -p acceptance-tests/${PRD_ID}/runs/evidence

Step 3: Execute Each Criterion

For each acceptance criterion, spawn a Sonnet sub-agent:

Task(
    subagent_type="general-purpose",
    model="sonnet",  # Sonnet for capability
    description=f"Execute acceptance test: {criterion_id}",
    prompt=f"""Execute acceptance test for criterion: {criterion_id}

## Test Definition
{criterion_yaml_content}

## Your Mission
1. Execute each step in order
2. Capture evidence as specified
3. Report PASS or FAIL with details

## Tools Available
- **Browser tests**: Use chrome-devtools MCP
  - mcp__chrome-devtools__navigate
  - mcp__chrome-devtools__click
  - mcp__chrome-devtools__fill
  - mcp__chrome-devtools__screenshot
  - mcp__chrome-devtools__evaluate (for assertions)

- **API tests**: Use Bash with curl
  - curl for HTTP requests
  - jq for JSON parsing

## Evidence Capture
Save all evidence to: acceptance-tests/{prd_id}/runs/evidence/
- Screenshots: {criterion_id}-{{description}}.png
- API responses: {criterion_id}-{{description}}.json

## Return Format
Return a structured result:
```yaml
criterion_id: {criterion_id}
status: PASS | FAIL
duration_seconds: <number>
evidence_files:
  - path: "evidence/..."
    description: "..."
steps_executed:
  - step_id: step-1
    status: PASS | FAIL
    details: "..."
failure_reason: null | "Description of why it failed"
actual_outcome: "What actually happened"

""" )

### Step 4: Aggregate Results

Collect results from all sub-agents:
- Count PASS/FAIL/SKIP
- Gather evidence paths
- Identify blocking failures

### Step 5: Generate Report

Create comprehensive markdown report at `acceptance-tests/{PRD_ID}/runs/{TIMESTAMP}.md`.

Report structure:
1. **Header**: PRD info, timestamp, environment
2. **Summary table**: PASS/FAIL counts with percentages
3. **Results by criterion**: Detailed results for each test
4. **What Works**: List of passing functionality
5. **What Doesn't Work**: List of failures
6. **Blocking Issues**: Criteria that must pass before closure
7. **Recommendations**: Actionable next steps
8. **Evidence Files**: Table linking evidence to criteria

See `references/report-template.md` for complete template.

### Step 6: Return Summary

Return concise summary to caller:
```yaml
prd_id: PRD-AUTH-001
overall_status: PASS | PARTIAL | FAIL
criteria_results:
  total: 5
  passed: 4
  failed: 1
  skipped: 0
blocking_failures:
  - AC-password-reset-complete
report_path: acceptance-tests/PRD-AUTH-001/runs/2026-01-24T10-30-00Z.md

Browser Test Execution

For validation_type: browser tests, use chrome-devtools MCP:

Action Mapping

Example Browser Execution

# Step: navigate to /login
mcp__chrome-devtools__navigate(url="http://localhost:3000/login")

# Step: fill email input
mcp__chrome-devtools__fill(selector="[data-testid='email-input']", value="test@example.com")

# Step: click button
mcp__chrome-devtools__click(selector="[data-testid='login-button']")

# Step: assert_visible with text check
result = mcp__chrome-devtools__evaluate(
    expression="document.querySelector('[data-testid=\"user-greeting\"]')?.textContent"
)
assert "Welcome" in result

# Step: capture screenshot
mcp__chrome-devtools__screenshot(path="acceptance-tests/PRD-AUTH-001/runs/evidence/AC-user-login-success.png")

API Test Execution

For validation_type: api tests, use Bash with curl:

Example API Execution

# Make request and capture response
curl -s -w "\n%{http_code}" \
  -X GET "http://localhost:8000/api/user/profile" \
  -H "Authorization: Bearer invalid-token" \
  -H "Content-Type: application/json" \
  > /tmp/response.txt

# Extract status code (last line)
STATUS=$(tail -n1 /tmp/response.txt)

# Extract body (all but last line)
BODY=$(sed '$d' /tmp/response.txt)

# Assert status
[ "$STATUS" = "401" ] && echo "PASS: Status is 401" || echo "FAIL: Expected 401, got $STATUS"

# Assert JSON field
echo "$BODY" | jq -e '.error == "Unauthorized"' && echo "PASS: Error field correct"

# Save evidence
echo "$BODY" | jq . > acceptance-tests/PRD-AUTH-001/runs/evidence/AC-api-auth-response.json

Failure Analysis

When a criterion fails, the sub-agent should provide:

1. Which step failed: Identify exact step ID
2. Expected vs Actual: What should have happened vs what did
3. Root cause hypothesis: Best guess at why it failed
4. Recommended action: Specific fix suggestion

Example failure analysis:

### Failure Analysis
- **Step 3 failed**: navigate to /reset-password/:token returned 404
- **Expected**: Page loads with password reset form
- **Actual**: 404 Not Found error page
- **Root Cause Hypothesis**: The `/reset-password/:token` route is not implemented in the frontend router
- **Recommended Action**:
  1. Add route in `app/router.tsx` for `/reset-password/:token`
  2. Create `ResetPasswordPage` component
  3. Re-run this acceptance test

Integration with Validation Agent

The validation-test-agent invokes this skill for --mode=e2e:

# In validation-test-agent routing logic
if mode == "e2e" and prd:
    if Path(f"acceptance-tests/{prd}").exists():
        Skill("acceptance-test-runner", args=f"--prd={prd} --task_id={task_id}")
    else:
        warn(f"No acceptance tests for {prd} - falling back to generic E2E")

Additional Resources

Reference Files

• references/report-template.md - Complete validation report template
• references/chrome-mcp-actions.md - Browser automation patterns

Example Files

• examples/validation-report.md - Sample validation report

Error Handling

No Acceptance Tests Found

ERROR: No acceptance tests found for PRD-AUTH-001
Path checked: acceptance-tests/PRD-AUTH-001/
Action: Generate acceptance tests first with:
  Skill("acceptance-test-writer", args="--prd=PRD-AUTH-001 --source=<prd-path>")

Browser Not Available

ERROR: Chrome DevTools MCP not available
Action: Ensure Chrome is running with debugging enabled
  or use mcp__chrome-devtools__launch_browser first

Sub-Agent Timeout

WARN: Criterion AC-user-login timed out after 60s
Status: SKIP
Action: Check if application is running and responsive

Checklist Before Returning

• All criteria executed (or marked as SKIP with reason)
• Evidence captured for each criterion
• Report generated at acceptance-tests/{PRD_ID}/runs/{TIMESTAMP}.md
• Summary returned to caller with overall status
• Blocking failures clearly identified