aibot88 ultra-test — Phase 3.5
Project-level quality audit before /ultra-deliver. Auditors are orthogonal;
each writes a JSON gate result; the skill aggregates into .ultra/test-report.json.
This is not for running unit tests (that is /ultra-dev). This is the
pre-ship gate.
Prerequisites
task.listreturns ≥1 task withstatus=completed- Repo has test files (Step 0 pre-check)
Workflow
Step 0 — Pre-Execution
- Detect project type from config (
package.json/Cargo.toml/go.mod/pyproject.toml/ …) task.list { status: "completed" }→ confirm ≥1 task delivered- Find test files (suffix
.test.*,.spec.*,test_*.py,*_test.go)
Block with instructive error if any precheck fails.
Step 1 — Anti-Pattern Detection
Detect fake / meaningless tests that waste CI time.
| Pattern | Example | Severity |
|---|---|---|
| Tautology | assert True, expect(true).toBe(true) |
CRITICAL |
| Empty test | function body is just pass / {} |
CRITICAL |
| Core-logic mock | jest.mock() on domain/service/state files |
CRITICAL |
Use Grep with language-appropriate regex; count matches per file.
Gate: any CRITICAL match → fail.
Step 2 — Coverage Gap Analysis
Find exported functions / classes with zero test references.
- Enumerate exports in
src/(or language equivalent) - For each symbol, grep test files for its name
- Bucket:
- HIGH: core business logic untested
- MEDIUM: utility untested
- LOW: config/constant untested
Output: .ultra/docs/test-coverage-gaps.md
Gate: any HIGH → fail.
Step 2.5 — Wiring Verification
Detect orphan code — files that exist and pass tests but are not reachable.
- Enumerate exported symbols in source files
- For each export, grep non-test source for imports
- Report exports with 0 non-test imports as orphaned
Also check boundary wiring:
- Component → API (fetch/axios targets a real route)
- API → DB (handler imports a real client)
- Form → handler (onSubmit is not
() => {}) - State → render (state appears in JSX/template)
Stub detection (Level-2 substantive):
- Functions returning empty
[]/{}without real work console.log-only bodies- Handlers calling only
e.preventDefault() - Components returning only
<div>Placeholder</div>
Output: append to test-coverage-gaps.md under ## Wiring Gaps.
Gate: any HIGH orphan or stub → fail.
Step 3 — E2E Testing (conditional)
Trigger: project has web UI or API endpoints.
- Start dev server (detect
scripts.dev/scripts.start) - Navigate key pages/endpoints
- Verify elements render; check console errors
- Exercise primary user flows
Method per runtime:
- Claude:
mcp__claude-in-chrome__*or Playwright via Bash - Others: Playwright via
npx playwright(CLI portable)
Gate: critical pages fail / major errors → fail.
Step 4 — Performance (conditional, frontend only)
Lighthouse on dev server. Core Web Vitals gates:
- LCP < 2.5s
- INP < 200ms
- CLS < 0.1
Gate: any metric above threshold → fail.
Step 5 — Security Audit
Dependency vulnerability scan via project's native tool:
npm audit --json/pnpm audit --json/yarn audit --jsoncargo audit --jsonpip-audit --format=jsongovulncheck ./...
Severity:
- Critical/High → fail
- Medium → warn
- Low → info
Step 6 — Auto-Fix Loop
If any gate failed, loop (max 5 attempts):
- Read
blocking_issues - Fix what is fixable without user input:
- Coverage Gap → write missing tests
- Anti-Pattern → rewrite fake tests
- E2E error → fix the code
- Performance → code-level optimizations (lazy, splitting, memoization)
- External blockers (upstream CVE, breaking dep bump) → break, report, ask user
- If all gates pass → exit loop
If max reached → surface remaining issues; passed=false.
Step 7 — Persist .ultra/test-report.json
{
"timestamp": "<ISO8601>",
"git_commit": "<HEAD>",
"passed": true,
"run_count": 1,
"gates": {
"anti_pattern": { "passed": true, "critical": 0, "warning": 1 },
"coverage_gaps": { "passed": true, "high": 0, "medium": 3, "low": 5 },
"wiring": { "passed": true, "orphans": 0, "stubs": 0 },
"e2e": { "passed": true, "skipped": false },
"performance": { "passed": true, "lcp": 1.8, "inp": 150, "cls": 0.05 },
"security": { "passed": true, "critical": 0, "high": 0, "medium": 2 }
},
"blocking_issues": []
}
Rules:
- File exists → increment
run_count passed = trueiff all gates.passed === trueblocking_issueslists human-readable strings for any fail
Step 8 — Report
Print a compact summary (emoji-free, ≤20 lines). Suggest /ultra-deliver only
when passed=true.
Quality Gates — summary
| Gate | Requirement |
|---|---|
| Anti-Pattern | 0 critical |
| Coverage Gaps | 0 HIGH |
| Wiring | 0 HIGH orphan, 0 HIGH stub |
| E2E | all flows pass (if applicable) |
| Performance | all Core Web Vitals (if frontend) |
| Security | 0 critical/high |
MCP → CLI fallback matrix
| Purpose | MCP tool | CLI fallback |
|---|---|---|
| Confirm ≥1 completed task | task.list { status: "completed" } |
ultra-tools task list --status completed |
| Confirm risky auto-fix | ask.question |
Claude: AskUserQuestion; CLI: ultra-tools ask --question … |
What this skill DOES NOT do
- Does NOT run unit tests (that is
/ultra-dev) - Does NOT mutate state.db (auditing is read-only against state)
- Does NOT release or tag (that is
/ultra-deliver)
Integration
| Input | Source + test files, state.db (read-only) |
| Output | .ultra/test-report.json, .ultra/docs/test-coverage-gaps.md |
| Next | /ultra-deliver (only when passed=true) |