Post Dev Verification

Post-Development Verification

Automated full-stack quality verification after development. Real execution by default -- mock is the last resort. Deliverability is judged by external calls (HTTP requests, CLI invocations, browser interactions), not by internal function calls passing in isolation.

Core Philosophy: Real Execution First

Default realism level is L2: internal services run for real, only uncontrollable external dependencies (third-party APIs, paid services) may be mocked.

Downgrade signals (auto-detected from user intent):

• "快速验证" / "只测逻辑" / "mock就行" -> L0
• Pure function / utility library with no I/O -> L0
• Local environment cannot start service -> L1 (report reason)
• "生产级别" / "全面测试" / "验收测试" -> L3

Realism level definitions:

Workflow

Execute phases sequentially. Each phase produces required artifacts for the next.

Phase 0: Environment Awareness
  v
Phase 1: Test Design (with anti-pattern scan)
  v
Phase 2: Execution & Evaluation
  v
Phase 3: Feedback & Fix Loop (if gates fail)
  v
Phase 4: Validation & Output

Safety Requirements

This skill starts services, runs migrations, and makes network calls. Before execution:

1. Run in isolation -- use a test/sandbox environment, never production systems
2. Use test credentials only -- test accounts, test API keys, environment-variable tokens; never supply production secrets
3. Phase 0 first -- review the Environment Report before allowing execution phases (Phase 2+) to understand what will be accessed
4. Scoped side effects -- all service starts, DB migrations, test data seeding, and cleanup are limited to the test environment
5. User control -- the user can downgrade realism level (e.g., "快速验证" for L0) to reduce scope at any time

Phase 0: Environment Awareness

Gather project context and determine feasibility before designing tests.

0.1 Project Analysis

Identify:

• Language & Framework: from config files (package.json, pyproject.toml, go.mod, etc.)
• Project type: monorepo / microservice / full-stack / library / CLI tool
• Monorepo scope (if applicable): identify which packages/services are affected by the current change
• Test runner: detect existing test framework (pytest, vitest, jest, go test, etc.)
• Coverage tool: detect coverage support (pytest-cov, vitest --coverage, nyc, etc.)
• Build/start commands: from scripts, Docker configs, Makefiles

0.2 Dependency Mapping

For each dependency service, classify:

• Controllable (self-hosted, has test env) -> must run real at L2+
• Uncontrollable (third-party, paid, no sandbox) -> acceptable to mock

0.3 Realism Level Decision

1. Check user intent signals -> override default if found
2. Check test target type -> pure functions auto-downgrade to L0
3. Assess local environment feasibility -> downgrade if services can't start
4. Record decision with rationale in the environment report

0.4 Environment Feasibility Check

Verify:

• Required tools installed (runtime, package manager, Docker if needed)
• Dependency services available (databases, caches, message queues)
• Target service can start locally
• Environment consistency: Docker/config matches production; environment variables and config files are complete; dependency service versions align with deployment target

Output: Environment Report -- language, framework, test runner, realism level, service availability, any blockers, consistency gaps.

Phase 1: Test Design

Design test scenarios systematically using the test taxonomy. Do NOT write test code before completing analysis.

1.1 Pre-Analysis (Anti Leap-to-Code)

Before writing any test code, complete:

• Code structure analysis: control flow, function signatures, module dependencies
• Dependency analysis: external services, databases, API contracts involved
• Constraint analysis: business rules, data invariants, input/output contracts

If test code references modules/imports not in the actual codebase -> hallucinated dependency. Remove and use actual project references.

1.2 Scenario Identification

Map each requirement to test scenarios:

• Functional scenarios (one per acceptance criterion minimum)
• Integration scenarios (cross-module, data flow, API contract)
• Error/exception scenarios (all error handling branches)
• End-to-end business flows (complete user journeys that traverse multiple steps/services -- e.g., "register -> verify email -> login -> place order -> pay -> receive confirmation"; "admin creates user -> assigns role -> user logs in with correct permissions"). Each identified critical business flow MUST have at least one E2E test validated through external calls.

Anti-pattern check: If >80% of scenarios are happy path -> Happy Path Obsession detected. Add error, boundary, and exception scenarios until >=30% target error scenarios.

1.3 Apply Test Taxonomy

For each scenario, apply the appropriate taxonomy category. Load detailed guidance from references/test-taxonomy.md when needed.

Coverage rule: Each of the 3 categories MUST have >=1 test. Taxonomy coverage = 100% is a hard gate.

1.4 Boundary Value Coverage

For every input parameter, identify and cover:

• Numeric: min-1, min, min+1, typical, max-1, max, max+1
• String/Collection: empty, single item, typical, max, exceeds max, special chars, Unicode
• Optional: not provided (default), explicit null/undefined, explicit empty

1.5 Real vs Mock Classification

Classify each scenario based on realism level:

• L2+: Internal service interactions -> must run real
• Any level: Uncontrollable external deps -> acceptable to mock (use realistic stubs)
• Track: test realism ratio = real tests / total tests >= 70% (hard gate at L2)

1.6 Visible/Hidden Test Separation

After designing all scenarios:

• Select 30-40% as hidden tests (prioritize edge cases, adversarial inputs, error scenarios)
• Hidden tests are NOT exposed during fix loop -- reserved for Phase 4 validation
• Record the split in the test plan

1.7 Anti-Pattern Scan

Run the pre-execution checklist. Load detailed guidance from references/anti-patterns.md when needed.

Rule: Each test name MUST describe the specific behavior being tested (e.g., test_submit_empty_form_returns_422). Each test MUST link to the requirement it validates.

Fix all detected anti-patterns before proceeding to Phase 2.

Phase 1 Output

• Test plan with all scenarios (tagged: MFT/INV/DIR, real/mock, visible/hidden)
• Boundary value coverage matrix
• Anti-pattern scan results (all clear)

Phase 2: Execution & Evaluation

2.1 Environment Preparation

Start services in dependency order with health checks:

for each service in dependency_order:
    start service
    wait_for_health_check (port/ping/readiness endpoint)
    if health_check fails:
        report blocker, downgrade realism level

Prepare test data using project's existing seed/migration mechanisms when available.

System boot validation -- before running any tests, verify the system itself is deliverable:

• The target service starts successfully from a clean state (no cached artifacts)
• All health/readiness endpoints return healthy
• Startup logs contain no ERROR-level entries (WARNs are logged for review)
• Required configuration and environment variables are complete
• If any boot validation fails -> this is a delivery blocker, report immediately

2.2 Test Execution

Run tests with coverage enabled. The test suite MUST include an E2E layer validated through external calls (HTTP requests to running services, CLI invocations, or browser interactions -- not internal function imports). Load execution templates from references/real-e2e-templates.md when designing this layer.

• Detect and use the project's native test runner and coverage tool
• Execute all visible tests (hidden tests reserved)
• E2E layer: exercise all identified critical business flows through external interfaces. If the project exposes an HTTP API, at minimum send real HTTP requests to each endpoint. If CLI, invoke real commands. If UI, drive real browser interactions.
• Capture: pass/fail per test, assertion output, error messages, coverage data

2.3 Metrics Collection

Compute all 4 layers of metrics. Load detailed definitions from references/metrics.md when needed.

Design Quality (computed after test design, before execution):

Execution Quality (computed after test run):

Delivery Quality (computed from test results):

Iteration Efficiency (computed during fix loop):

Phase 3: Feedback & Fix Loop

Triggered when hard gate metrics are not met.

3.1 Generate Feedback Report

Structure the report as JSON. Load schema from references/feedback-schema.md when needed.

Key sections:

• round: current iteration number
• project_context: language, framework, test runner, realism level
• metrics: all 4 layers with pass/fail per metric
• gate_result: "pass" | "fix_and_retry" | "stop_and_report"
• failures_grouped: cluster by root cause, each with affected tests + fix direction
• fix_history: timeline of each round's metrics before/after, fix description, newly passing/failing
• hidden_tests: total count, run status, results (null during fix loop, populated in Phase 4)
• anti_patterns_detected: names of anti-patterns found in current test suite
• next_action: what the Agent should do next

3.2 Fix Prioritization

Address the largest failure cluster first (most affected tests) -- highest probability of improving overall pass rate.

3.3 Convergence Control -- Triple Exit Conditions

3.4 Fix Loop Cycle

while round <= 5 AND not converged_stop AND not regression_stop:
    read feedback report -> identify largest failure cluster
    apply targeted fix to code/tests
    re-run full test suite (or failed-only if convergence is high)
    compute new metrics
    generate new feedback report
    check exit conditions
    record iteration in fix history

Phase 3 Output

• Fix history timeline (round, metrics before/after, fix description, newly passing/failing)
• Final feedback report with gate_result

Phase 4: Validation & Output

4.1 Hidden Test Validation

Run ALL hidden tests (not exposed during fix loop):

• If hidden tests pass -> validates that fixes are genuine, not overfitted
• If hidden tests fail -> report failures, do NOT re-enter fix loop

4.2 Quality Report

Generate final quality report:

• Verdict: "PASS" (all hard gates met) or "FAIL" (hard gates not met) with specific reasons
• All 4 layers of metrics with pass/fail status
• Fix history timeline
• Hidden test validation results
• Anti-pattern scan summary
• Recommendations for improvement

4.3 Reusable Test Scripts

Output test scripts that can run independently in CI/CD:

• Follow project's native test format and directory conventions
• Include run instructions: prerequisites, service startup, how to run, expected output
• Include environment documentation: services needed, config required, ports

Quality Gate Quick Reference

Hard Gates (MUST pass -- blocks delivery)

Soft Targets (SHOULD pass -- reported, does not block unless configured)

References

Load these files as needed during the workflow:

• references/metrics.md -- Complete definitions for all 14 metrics: calculation formulas, threshold rationale, and what it means when a metric is not met. Load when computing or interpreting metrics.

• references/test-taxonomy.md -- Detailed guidance for MFT/INV/DIR test categories with pseudocode examples, plus systematic boundary value analysis methods. Load during Phase 1 test design.

• references/feedback-schema.md -- Full JSON Schema for feedback reports with a populated example. Load when generating feedback reports in Phase 3.

• references/anti-patterns.md -- Detailed detection methods and fix strategies for all 5 anti-patterns: Happy Path Obsession, Weak Assertions, Leap-to-Code, Hallucinated Dependencies, Missing Traceability. Load during Phase 1 anti-pattern scan or Phase 3 feedback.

• references/real-e2e-templates.md -- Environment preparation scripts and real E2E test templates for HTTP API, CLI tools, and Browser automation. Load during Phase 2 environment preparation and test execution.