Quality Assurance

Quality Assurance

Overview

This skill implements the Verification (VER) and Validation (VAL) process areas from the CMMI-based SDLC prescription.

Core principle: Verification ≠ Validation. Tests prove you built it correctly (VER). Users prove you built the right thing (VAL). Both required at Level 3.

Critical distinction:

• Verification: "Did we build the product right?" (tests, reviews, inspections)
• Validation: "Did we build the right product?" (user acceptance, stakeholder approval)

Reference: See docs/sdlc-prescription-cmmi-levels-2-4.md Section 3.3 for complete VER/VAL policy.

When to Use

Use this skill when:

• Deciding test strategy or coverage requirements
• Code reviews ineffective ("LGTM" rubber stamps)
• Pressure to skip tests ("we'll add them later")
• Tests pass but customers report bugs (VER without VAL)
• Same defects recurring (no root cause analysis)
• Manual testing taking days (ice cream cone anti-pattern)
• Unclear what "quality" means for your project level

Do NOT use for:

• Specific test framework details → Use domain skills (python-engineering, web-backend)
• E2E/performance/chaos engineering → Use ordis-quality-engineering
• Production monitoring → Use platform-integration

Quick Reference

Verification vs Validation: The Critical Distinction

Verification (VER) - "Built Correctly"

What: Ensuring product meets specifications and requirements

How: Testing, code review, static analysis, inspections

Who: Development team (internal)

When: Throughout development, before release

Level 3 Requirements:

• Test coverage >70% for critical paths
• Peer review required for all changes
• Automated tests in CI pipeline

Example: Unit tests pass, integration tests pass, code reviewed

Validation (VAL) - "Right Thing Built"

What: Ensuring product meets user needs and solves actual problems

How: User acceptance testing (UAT), stakeholder demos, beta testing

Who: End users, stakeholders (external to dev team)

When: End of iteration, before production release

Level 3 Requirements:

• Stakeholder sign-off on acceptance criteria
• UAT with representative users
• Demo to product owner for approval

Example: Users confirm feature solves their problem, stakeholders approve for release

Why Both Matter

Level 3 mandate: Both VER and VAL required before production release.

Level-Based QA Requirements

Level 2: Managed

VER Requirements:

• Basic test coverage (>50% for critical paths)
• Peer review recommended (not enforced)
• Manual testing acceptable

VAL Requirements:

• Product owner approval
• Informal stakeholder feedback

Work Products:

• Test results (pass/fail)
• Review notes (PR comments)

Quality Criteria:

• Critical functionality tested
• Stakeholder aware of release

Audit Trail:

• Test runs logged
• Approval emails/messages

Level 3: Defined

VER Requirements:

• Required test coverage >70% for critical paths
• Mandatory peer review (2+ reviewers, platform-enforced)
• Automated testing in CI
• Code review checklist used
• Test strategy documented

VAL Requirements:

• UAT with representative users required
• Formal stakeholder acceptance criteria
• Demo to product owner mandatory
• Validation documented (sign-off)

Work Products:

• Test strategy document
• Test coverage reports
• Review effectiveness metrics
• UAT plan and results
• Stakeholder acceptance sign-off

Quality Criteria:

• Coverage targets met (>70%)
• Review finding rate 20-40% (detects real issues)
• Stakeholder approval documented
• Defect escape rate tracked

Audit Trail:

• All tests tracked in CI
• Review approvals in platform
• UAT sign-off with dates
• Defect metrics dashboard

Level 4: Quantitatively Managed

Statistical Practices:

• Defect prediction models (based on complexity, churn)
• Review effectiveness statistical control (control charts)
• Test coverage trends with baselines
• Defect escape rate within statistical limits

Quantitative Work Products:

• Statistical process control charts (defect injection, escape rates)
• Predictive models for quality
• Cp/Cpk analysis for test processes

Quality Criteria:

• Defect density <0.5 per KLOC (baseline established)
• Review finding rate within control limits (20-40%)
• Test coverage stable >80% with minimal variation

Audit Trail:

• Historical quality data with statistical analysis
• Prediction vs actual defect counts
• Process capability indices

Exception Protocol: Shipping Without Tests

CRITICAL: "Tests later" = tests never (documented historical pattern)

When Shipping Without Tests is NEVER Acceptable

Level 3 projects - Absolute requirements:

• Critical user-facing features
• Security-sensitive code (auth, payments, PII)
• Regulatory/compliance features
• Data migration or modification
• Core business logic

Rationale: Risk too high, rework too expensive, reputation damage too severe

Emergency Exception Process (TEST-HOTFIX)

When: Production outage, immediate fix needed, no time for full test suite

Level 3 Requirements:

1. Fix the emergency (restore service)
2. Document in issue tracker with "TEST-HOTFIX" label
3. Write tests within 48 hours (retrospective testing mandatory)
4. Create ticket for proper fix if hotfix is hack
5. RCA for why hotfix needed (prevent future)

Frequency Limit: >5 TEST-HOTFIXes per month = systemic problem requiring process audit

Violation: Skipping retrospective tests = QA failure, escalate to engineering manager

Risk-Based Minimal Testing (When Must Ship)

If absolutely must ship without full coverage:

1. Critical path only: Test happy path + 1-2 critical error cases
2. Feature flag: Deploy disabled, enable after testing
   • Maximum duration flagged: 7 days before full test suite required
   • Flagged features count toward TEST-HOTFIX frequency limit
   • Must have validation plan with timeline before deploying flagged
3. Beta rollout: Ship to 5-10% users, monitor, expand
4. Demo ≠ Production: Demo to stakeholders, don't enable for all users
   • Maximum demo-only duration: 2 sprints
   • After demo: either release to production (with UAT) or cancel feature
   • "Perpetual demo" is validation theater anti-pattern
5. Manual acceptance test: At minimum, stakeholder uses feature live

Retrospective required: Within 7 days, answer "Why no tests?" and address root cause

Enforcement: Violations escalate to engineering manager, process audit if patterns emerge

Anti-Patterns and Red Flags

Test Last

Detection: Tests written after code (or not at all), "We'll add tests later"

Red Flags:

• PR without tests for new functionality
• Test coverage declining sprint-over-sprint
• "Too busy to write tests"
• Tests added only when bugs found

Why it fails: "Later" never comes, test debt accumulates, bugs reach production, rework costs 10-100x more

Counter: TDD requirement (Level 3 can waive, but must justify). Tests = part of "done", not optional.

Rubber Stamp Reviews

Detection: Code reviews <5 minutes, "LGTM" without specific feedback, defects escaping to production

Red Flags:

• Review approved within minutes of PR creation
• No comments or only style nitpicks
• Reviewer didn't pull code or run it
• Same bugs recurring that reviews should have caught

Why it fails: Social pressure not to block > quality, reviewers fear being "difficult", no accountability

Counter: Review metrics (finding rate should be 20-40%), reviewer accountability, psychological safety

Ice Cream Cone (Inverted Test Pyramid)

Detection: Mostly manual E2E tests, few unit tests, regression testing takes days

Red Flags:

• 50% of testing time is manual

• Regression suite takes >4 hours
• Most tests are UI/E2E (slow, brittle)
• "Can't automate, need manual QA"

Why it fails: Doesn't scale, slow feedback, expensive to maintain, brittle tests

Counter: Test pyramid economics, migration to unit-heavy strategy, ROI calculation

Defect Whack-a-Mole

Detection: Same bugs recurring in different places, no pattern analysis, firefighting constantly

Red Flags:

• Similar bugs in different modules (copy-paste errors)
• Defects closed without RCA
• "Fix it quick, no time to investigate"
• Bug fixes create new bugs (ripple effects)

Why it fails: Treats symptoms not causes, waste effort on recurring issues, no learning

Counter: RCA requirement (Level 3 mandatory for recurring defects), defect pattern analysis

Validation Theater

Detection: Stakeholders "approve" without actually using system, checkbox exercise

Red Flags:

• UAT sign-off in <1 hour (didn't actually test)
• Stakeholders approve without touching the system
• Demo only (no hands-on validation)
• Rubber stamp "looks good" without criteria
• Product owner used as "representative user" instead of actual end users
• Feature in perpetual demo mode (>2 sprints without production release or cancellation)

Why it fails: False confidence, issues found in production, customer dissatisfaction

Counter:

• Hands-on UAT requirement
• Level 3 requires at least 2 actual end users for UAT (not proxies)
• Product owner is NOT a representative user (unless they use the product daily)
• Exception: Internal tools where team members are actual users
• Acceptance criteria verification
• Time requirement (min 1 day for meaningful validation)
• Demo-only maximum: 2 sprints before release or cancel decision

Reference Sheets

The following reference sheets provide detailed guidance for specific QA domains. Load them on-demand when needed.

1. Testing Practices

When to use: Deciding test strategy, coverage requirements, test pyramid, TDD

→ See testing-practices.md

Covers:

• Test pyramid (unit, integration, E2E) with economics
• Coverage criteria by project level
• Test-driven development (TDD) process
• Test types (smoke, regression, acceptance)
• Migration from manual to automated (ice cream cone → pyramid)
• Anti-patterns: Test Last, Over-Mocking, Flaky Tests

2. Peer Reviews

When to use: Code reviews ineffective, rubber-stamp approvals, unclear reviewer responsibilities

→ See peer-reviews.md

Covers:

• Review checklist (functionality, tests, design, security)
• Social dynamics playbook (giving critical feedback safely)
• Reviewer accountability and responsibilities
• Review metrics (effectiveness, turnaround time, finding rate)
• Review taxonomy (depth varies by change type: hotfix vs feature)
• Anti-patterns: Rubber Stamp, Bikeshedding, Review Backlog

3. Validation with Stakeholders

When to use: Planning UAT, stakeholder acceptance, beta testing, demo preparation

→ See validation-with-stakeholders.md

Covers:

• UAT process and planning
• Acceptance criteria definition (INVEST)
• Stakeholder identification and management
• Demo vs hands-on validation
• Beta rollout strategies
• Anti-patterns: Validation Theater, Demo-Only, Proxy Users

4. Defect Management

When to use: Bugs recurring, defect triage, root cause analysis, prevention

→ See defect-management.md

Covers:

• Defect classification (severity, recurrence, root cause)
• Root cause analysis (5 Whys, fishbone diagram, fault tree)
• Defect prevention over detection
• Level 3 requirement: RCA for recurring defects
• Defect metrics (escape rate, density, resolution time)
• Anti-patterns: Whack-a-Mole, Symptom Fixes, No RCA

5. QA Metrics

When to use: Measuring quality effectiveness, tracking improvement, justifying QA investment

→ See qa-metrics.md

Covers:

• Defect escape rate (bugs found post-release / total bugs)
• Review effectiveness (finding rate: bugs in review / total bugs)
• Test automation ROI
• Coverage trends and targets
• Level 4 statistical process control

6. Level 2→3→4 Scaling

When to use: Understanding appropriate QA rigor for project tier

→ See level-scaling.md

Covers:

• Level 2 baseline QA practices
• Level 3 organizational QA standards
• Level 4 statistical quality control
• Escalation criteria (when to increase rigor)
• De-escalation criteria (when rigor is overkill)

Common Mistakes

Integration with Other Skills

Real-World Impact

Without this skill: Teams experience:

• VER without VAL (tests pass, customers unhappy)
• Test debt accumulating ("later" never comes)
• Rubber stamp reviews (LGTM without reading)
• Same defects recurring (no RCA)
• Ice cream cone (slow manual E2E tests)

With this skill: Teams achieve:

• Both VER and VAL (quality gate before production)
• Tests written alongside code (TDD culture)
• Effective reviews (20-40% finding rate)
• Defect prevention through RCA
• Test pyramid (fast feedback, low maintenance)

Next Steps

1. Determine project level: Check CLAUDE.md or ask user for CMMI target level (default: Level 3)
2. Identify situation: Use Quick Reference table to find relevant reference sheet
3. Load reference sheet: Read detailed guidance for specific domain
4. Enforce VER+VAL: Level 3 requires both verification and validation - no exceptions
5. Apply frameworks: Use systematic evaluation (test pyramid economics, review metrics, RCA methods)
6. Counter anti-patterns: Watch for test-last, rubber stamps, ice cream cone, whack-a-mole
7. Measure effectiveness: Establish baselines, track defect escape rate and review finding rate

Remember: Verification proves you built it correctly. Validation proves you built the right thing. You need BOTH.