jmagly UAT Execute
Execute a UAT plan against live MCP connections. Runs each phase sequentially, tracks pass/fail per test, files issues for failures, and always runs cleanup.
Usage
# Execute a UAT plan
/uat-execute .aiwg/testing/uat/plan-gitea-20260227.md
# Dry run — show what would be executed without making calls
/uat-execute .aiwg/testing/uat/plan.md --dry-run
# Skip issue filing
/uat-execute .aiwg/testing/uat/plan.md --no-issues
# Resume from a specific phase (after fixing a blocker)
/uat-execute .aiwg/testing/uat/plan.md --resume-from 3
# Override execution mode (run only quick smoke subset)
/uat-execute .aiwg/testing/uat/plan.md --execution-mode quick
Parameters
plan_path (required)
Path to the UAT plan document generated by /uat-generate.
--mode (default: mcp)
Test mode. Must match the plan's mode. Currently only mcp is supported.
--execution-mode (optional)
Override the plan's execution mode. Useful for running a quick smoke test from a full plan.
--dry-run (optional)
Parse the plan and show what would be executed without making any MCP calls.
--no-issues (optional)
Disable automatic issue filing for failures. Results are still recorded.
--resume-from (optional)
Resume execution from a specific phase number. Useful after fixing a blocker found mid-execution. Previous phase results are loaded from the checkpoint file.
Workflow
Step 1: Parse Plan
Read the UAT plan and extract:
- Phase list with prerequisites
- Test cases per phase with MCP call specs
- Variable wiring requirements
- Negative test isolation flags
Parsing UAT Plan...
Plan: plan-gitea-20260227.md
Phases: 12
Test cases: 165
Negative tests: 32 (will be isolated)
Variables: 18 cross-phase references
Step 2: Preflight Check
Before executing tests, verify:
- MCP server(s) are reachable
- Required tools are available
- Authentication is valid
Preflight Check...
Server: gitea — Connected
Tools available: 78/78
Auth: Valid (user: roctibot)
Ready to execute.
Step 3: Execute Phases
For each phase in order:
═══════════════════════════════════════
Phase 3: Issue Tracking (12 tests)
═══════════════════════════════════════
P03-001: Create Issue (Happy Path)
Tool: mcp__gitea__create_issue
Parameters: {owner: "test-org", repo: "test-repo", title: "UAT Test Issue", body: "..."}
Response: {number: 1, title: "UAT Test Issue", state: "open"}
✅ Criterion 1: Returns issue with number field — PASS
✅ Criterion 2: title matches — PASS
✅ Criterion 3: state is "open" — PASS
Stored: ISSUE_INDEX = 1
Result: PASS (3/3)
P03-002: Get Issue by Index
Tool: mcp__gitea__get_issue_by_index
Parameters: {owner: "test-org", repo: "test-repo", index: 1}
Response: {number: 1, title: "UAT Test Issue", ...}
✅ Criterion 1: Returns issue matching ISSUE_INDEX — PASS
✅ Criterion 2: title is "UAT Test Issue" — PASS
Result: PASS (2/2)
P03-003: Create Issue — Missing Title (Negative)
[Isolation: single call]
Tool: mcp__gitea__create_issue
Parameters: {owner: "test-org", repo: "test-repo", body: "No title"}
Response: {error: "validation failed"}
✅ Criterion 1: Returns error — PASS
✅ Criterion 2: Error indicates missing title — PASS
Result: PASS (2/2)
Phase 3 Summary: 12/12 passed (0 failed, 0 skipped)
Duration: 45 seconds
Step 4: Handle Failures
On test failure:
- Record the failure with full details
- File an issue (unless
--no-issues) - Continue to next test (don't stop)
- If a stored variable was expected but not captured, mark dependent tests as
error
P04-007: Update Repository Description
Tool: mcp__gitea__update_repo
Parameters: {owner: "test-org", repo: "test-repo", description: "Updated"}
Response: {error: "not found", message: "repository not found"}
❌ Criterion 1: Returns updated repository — FAIL
Result: FAIL (0/1)
Issue filed: #415 "UAT: P04-007 — Update Repository returns not found"
Step 5: Cleanup Phase
The cleanup phase ALWAYS runs, even if earlier phases failed:
═══════════════════════════════════════
Phase 11: Cleanup (always runs)
═══════════════════════════════════════
P11-001: Delete Test Repository
Tool: mcp__gitea__delete_repo (if available)
...
P11-002: Delete Test Organization
...
Cleanup complete. 3/3 test entities removed.
Step 6: Save Results
Write results to .aiwg/testing/uat/results/ following the uat-result.yaml schema:
Results saved:
File: .aiwg/testing/uat/results/results-gitea-20260227-143022.yaml
Checkpoint: .aiwg/testing/uat/results/checkpoint-gitea-20260227.yaml
Generate report with:
/uat-report .aiwg/testing/uat/results/results-gitea-20260227-143022.yaml
Step 7: Summary
═══════════════════════════════════════════
UAT Execution Complete
═══════════════════════════════════════════
Plan: plan-gitea-20260227.md
Duration: 8 minutes 32 seconds
Results:
Total tests: 165
Passed: 159 (96.4%)
Failed: 4 (2.4%)
Skipped: 1 (0.6%)
Errors: 1 (0.6%)
Issues filed: 4
#415: P04-007 — Update Repository returns not found
#416: P06-003 — Wiki page creation timeout
#417: P08-002 — Release tag not created
#418: P09-011 — Search returns stale results
Coverage: 76/78 tools tested (97.4%)
Untested: mcp__gitea__cancel_repo_action_run (no active runs)
mcp__gitea__stop_stopwatch (no active stopwatch)
═══════════════════════════════════════════
Error Handling
Plan Not Found
Error: UAT plan not found at {path}
Generate a plan first:
/uat-generate --mode mcp
MCP Server Unavailable
Error: MCP server {name} is not reachable.
The UAT plan requires this server. Check your MCP configuration.
Retry with: /uat-execute {plan_path}
Checkpoint Recovery
If execution crashes mid-run, results are checkpointed per-phase:
Checkpoint detected: .aiwg/testing/uat/results/checkpoint-gitea-20260227.yaml
Phases completed: 0-5
Phases remaining: 6-11
Resume with: /uat-execute {plan_path} --resume-from 6
References
- Agent: @$AIWG_ROOT/agentic/code/addons/uat-mcp/agents/uat-executor.md
- Schema: @$AIWG_ROOT/agentic/code/addons/uat-mcp/schemas/uat-result.yaml
- Template: @$AIWG_ROOT/agentic/code/addons/uat-mcp/templates/uat-executor-guide.md