Validation Strategist

Validation Strategist

Purpose

Use this skill to design a multi-layer validation and verification strategy for any software artifact or release. The output is a structured plan that maps what to validate at each layer, which environments and evidence are required, and what gates must pass before progressing.

This skill is domain-generic. It must work for PRDs, SDDs, RFCs, ADRs, ACs, NFRs, architectures, OpenAPI specs, workflows, and release plans without embedding project-specific assumptions.

Core Distinction: Verification vs Validation

A complete strategy uses both. Verification gives confidence in correctness; validation gives confidence in correctness.

The Five Validation Layers

Design validation across these five layers, from narrowest to widest scope:

Layer 1 — Unit

• What: Individual functions, methods, classes in isolation
• Evidence: Unit tests, code coverage, linting, type checks
• Gates: All tests pass; coverage above threshold; no critical lints
• Metrics: Pass rate, coverage %, cyclomatic complexity

Layer 2 — Component

• What: Integration between modules within a service or package
• Evidence: Integration tests, contract tests, module-level mocks
• Gates: Component tests pass; contracts between modules verified
• Metrics: Integration pass rate, contract compliance

Layer 3 — Feature

• What: End-to-end behavior of a single feature or user story
• Evidence: Feature tests, API contract tests, UI automation, Given/When/Then scenarios
• Gates: Feature acceptance criteria met; all happy and sad paths covered
• Metrics: Feature test pass rate, scenario coverage, critical path coverage

Layer 4 — Application

• What: Multiple features working together across services
• Evidence: System tests, end-to-end tests, performance benchmarks, security scans
• Gates: Cross-feature flows work; NFRs met; security scan clean
• Metrics: System test pass rate, p99 latency, error rate, security findings

Layer 5 — Release Candidate

• What: The deliverable in a production-like environment
• Evidence: Staging deployment, smoke tests, manual QA, canary rollout, rollback test
• Gates: Rollback procedure verified; monitoring and alerting operational; feature flags active; business metrics within expected range
• Metrics: Deployment success rate, time-to-production-ready, rollback success rate

Layer Selection Principle

Always prefer the lowest layer that gives enough signal.

If Layer 1 unit tests prove correctness, do not require Layer 4 system tests for that specific check. Higher layers are slower, more expensive, and harder to debug. Reserve them for what lower layers cannot catch:

• Cross-service interactions → Layer 3–4
• User-facing flows → Layer 4–5
• Critical-path flows → Layer 5 (E2E reserved for flows where failure is catastrophic)

E2E Abuse Prevention

End-to-end tests are a limited resource. Use them only when:

• The flow spans multiple services or domains.
• The consequence of failure is severe (data loss, security breach, financial impact).
• Lower-layer tests cannot simulate the failure mode.

If a flow can be validated at Layer 3 or 4 with reliable automation, do not add a Layer 5 E2E test.

Validation Triggers and Environments

Entry and Exit Criteria per Layer

For each active layer, define:

Entry criteria — what must be true before running validation at this layer:

• Prior layers passed and gates closed.
• Required test artifacts exist and are passing.
• Environment is stable and isolated.

Exit criteria — what must be true to consider this layer passed:

• All gates passed with evidence attached.
• No open critical or high severity findings.
• Metrics within defined thresholds.

Release Readiness Checklist

At Layer 5 (Release Candidate), validate all of the following before promoting to production:

Input Artifact Mapping

Adapt the validation strategy based on what artifact is provided:

Validation Strategy Output Structure

# Validation Strategy: <Artifact Name>

## 1. Scope and Context
- Artifact type: <PRD / SDD / RFC / ADR / AC / NFR / OpenAPI / Workflow / Release Plan>
- Validation goal: <what we need to prove>
- Active layers: <which of the 5 layers apply>

## 2. Verification vs Validation Split
- Verification checks: <list>
- Validation checks: <list>

## 3. Layer-by-Layer Plan
| Layer | What to Validate | Evidence | Gates | Metrics |
| --- | --- | --- | --- | --- |

## 4. E2E Justification
- Critical flows requiring E2E: <list>
- Why lower layers are insufficient for each: <reason>

## 5. Environment and Trigger Map
| Trigger | Environment | Evidence |
| --- | --- | --- |

## 6. Entry/Exit Criteria
| Layer | Entry Criteria | Exit Criteria |
| --- | --- | --- |

## 7. Release Readiness (Layer 5 only)
| Check | Status | Notes |
| --- | --- | --- |
| Unit + Component gates closed | ✅/❌ | |
| Feature ACs verified | ✅/❌ | |
| NFRs met | ✅/❌ | |
| Smoke tests pass | ✅/❌ | |
| Rollback tested | ✅/❌ | |
| Monitoring active | ✅/❌ | |
| Go/no-go authorized | ✅/❌ | |

## 8. QA Flow Summary
<One-paragraph description of the end-to-end QA flow from commit to production>

Quality Bar

Before presenting the validation strategy, verify:

• Verification and validation are treated as distinct activities with separate outputs.
• The lowest-layer principle is applied: no layer used beyond what it can meaningfully prove.
• E2E tests are justified and limited to critical paths.
• All five layers are evaluated; inactive layers are explicitly excluded with rationale.
• Entry and exit criteria are concrete and evidence-attached.
• The strategy is generic: no vendor names, project names, or concrete technology names unless essential.

Troubleshooting

• User provides no artifact type: Ask for the artifact type or the primary deliverable being validated before designing the strategy.
• All five layers requested for a simple change: Apply the lowest-layer principle strictly. A one-line bug fix may only need Layer 1 evidence.
• E2E tests are the only evidence available: Flag this as a risk. E2E is slow, brittle, and expensive to maintain. Recommend building lower-layer test coverage.
• No defined NFRs or ACs: Treat this as a gap — the strategy should note that entry/exit criteria cannot be verified without defined thresholds.
• Conflicting evidence from different layers: Investigate the discrepancy. Lower-layer failures may indicate an environment issue; higher-layer failures always take priority.