Validation

Feature Validation Skill

Purpose

This skill provides systematic validation of implemented features, ensuring code quality, test coverage, performance, security, and requirement fulfillment before marking work complete.

When to Use

• After implementation and testing are complete
• Before creating pull request
• Before marking feature as done
• When verifying all acceptance criteria met
• Final quality gate before deployment

Validation Workflow

1. Code Quality Validation

Run Quality Checks:

# Format check (Black)
black --check src/ tests/

# Type checking (mypy)
mypy src/

# Linting (flake8, if configured)
flake8 src/ tests/

# All checks together
make lint  # If Makefile configured

Quality Checklist: Refer to quality-checklist.md for comprehensive review

Key Quality Metrics:

• All functions have type hints
• All public functions have docstrings (Google style)
• No files exceed 500 lines
• No lint errors or warnings
• Code formatted with Black
• Type checking passes with mypy
• No code duplication (DRY principle)
• Single responsibility principle followed

Automated Script:

# Use validation script
python scripts/run_checks.py --quality

Deliverable: Quality report with pass/fail

2. Test Coverage Validation

Run Tests with Coverage:

# Run all tests with coverage
pytest --cov=src --cov-report=html --cov-report=term-missing

# Check coverage threshold
pytest --cov=src --cov-fail-under=80

# View HTML coverage report
open htmlcov/index.html

Coverage Checklist:

• Overall coverage ≥ 80%
• Core business logic ≥ 90%
• Utilities and helpers ≥ 85%
• No critical paths untested
• All branches covered
• Edge cases tested
• Error conditions tested

Identify Coverage Gaps:

# Show untested lines
pytest --cov=src --cov-report=term-missing

# Generate detailed HTML report
pytest --cov=src --cov-report=html

Deliverable: Coverage report with gaps identified

3. Test Quality Validation

Review Test Suite:

• All tests passing
• No skipped tests (without justification)
• No flaky tests (intermittent failures)
• Tests run quickly (unit tests < 1 min)
• Tests are independent (no order dependency)
• Tests clean up after themselves
• Mock external dependencies properly
• Test names are clear and descriptive

Run Tests Multiple Times:

# Run tests 10 times to check for flaky tests
for i in {1..10}; do pytest || break; done

# Run in random order
pytest --random-order

Test Markers:

# Verify no slow tests in unit tests
pytest tests/unit/ -m "not slow"

# Run integration tests separately
pytest tests/integration/

Deliverable: Test quality assessment

4. Performance Validation

Performance Checklist: Refer to performance-benchmarks.md for target metrics

Key Performance Metrics:

• Response time < target (e.g., < 200ms for p95)
• Throughput meets requirements (e.g., 1000 req/s)
• Memory usage within bounds (e.g., < 100MB)
• CPU usage reasonable (e.g., < 50%)
• No memory leaks detected
• Database queries optimized (< 5 queries per operation)

Performance Testing:

# Run performance tests
pytest tests/performance/ -v

# Profile code
python -m cProfile -o profile.stats script.py
python -m pstats profile.stats

# Memory profiling
python -m memory_profiler script.py

Benchmark Against Requirements:

# Example performance test
def test_performance_requirement():
    """Verify operation meets performance requirement."""
    start = time.time()
    result = expensive_operation()
    duration = time.time() - start

    assert duration < 1.0, f"Took {duration}s, required < 1.0s"

Deliverable: Performance report with metrics

5. Security Validation

Security Checklist Review: Review security-checklist.md from analysis phase and verify:

Input Validation:

• All user inputs validated and sanitized
• SQL injection prevented (parameterized queries)
• Command injection prevented (no shell=True with user input)
• Path traversal prevented (sanitized file paths)
• XSS prevented (escaped output)

Authentication & Authorization:

• Authentication required for protected endpoints
• Authorization checks at every access point
• Session management secure
• Credentials not hardcoded

Data Protection:

• Sensitive data encrypted in transit
• Sensitive data encrypted at rest (if applicable)
• PII handling compliant
• Secrets in environment variables (not code)
• Error messages don't leak sensitive info

Dependency Security:

# Check for vulnerable dependencies
pip-audit

# Or use safety
safety check --json

# Check for outdated dependencies
pip list --outdated

Deliverable: Security validation report

6. Requirements Validation

Verify Acceptance Criteria: Review original requirements from analysis phase:

• All functional requirements implemented
• All acceptance criteria met
• User stories fulfilled
• Edge cases handled
• Error scenarios handled

Manual Testing:

# Test CLI (if applicable)
python -m src.tools.feature.main --help
python -m src.tools.feature.main create --name test

# Test with sample data
python -m src.tools.feature.main --input samples/test.json

# Test error cases
python -m src.tools.feature.main --invalid-option

Regression Testing:

• Existing functionality not broken
• No breaking changes to public APIs
• Backward compatibility maintained (if required)

Deliverable: Requirements validation checklist

7. Documentation Validation

Code Documentation:

• All public functions have docstrings
• Docstrings follow Google style
• Complex logic has inline comments
• Type hints present and accurate
• README updated (if applicable)

Technical Documentation:

• Architecture documented
• API contracts documented
• Configuration documented
• Setup instructions complete
• Known issues documented

User Documentation:

• Usage guide written (if applicable)
• Examples provided
• Troubleshooting guide included
• FAQ updated

CHANGELOG Update:

• Changes documented in CHANGELOG.md
• Version bumped appropriately
• Breaking changes highlighted

Deliverable: Documentation review checklist

8. Integration Validation

Integration Testing:

# Run integration tests
pytest tests/integration/ -v

# Test with real dependencies (in test environment)
pytest tests/integration/ --no-mock

Integration Checklist:

• Integrates correctly with existing code
• No circular dependencies
• Module imports work correctly
• Configuration loads correctly
• External services connect (if applicable)

End-to-End Testing:

# Test complete workflows
pytest tests/e2e/ -v

# Manual E2E testing
./scripts/manual_test.sh

Deliverable: Integration test report

9. Final Validation

Run Complete Validation Suite:

# Use automated validation script
python scripts/run_checks.py --all

# Or run individual checks
python scripts/run_checks.py --quality
python scripts/run_checks.py --tests
python scripts/run_checks.py --coverage
python scripts/run_checks.py --security

Pre-PR Checklist:

• All quality checks passing
• Test coverage ≥ 80%
• All tests passing
• Performance requirements met
• Security validated
• Requirements fulfilled
• Documentation complete
• Integration verified
• No known critical bugs

Create Validation Report:

# Validation Report: [Feature Name]

## Quality ✅
- Black: PASS
- mypy: PASS
- flake8: PASS (0 errors, 0 warnings)

## Testing ✅
- Unit tests: 45 passed
- Integration tests: 12 passed
- Coverage: 87% (target: 80%)

## Performance ✅
- Response time (p95): 145ms (target: < 200ms)
- Throughput: 1200 req/s (target: 1000 req/s)
- Memory usage: 75MB (target: < 100MB)

## Security ✅
- No vulnerable dependencies
- Input validation: Complete
- Secrets management: Secure

## Requirements ✅
- All acceptance criteria met
- No regressions detected

## Documentation ✅
- Code documentation: Complete
- Technical docs: Complete
- CHANGELOG: Updated

## Status: READY FOR PR ✅

Deliverable: Final validation report

Quality Standards

Code Quality Metrics

Complexity:

• Cyclomatic complexity < 10 per function
• Max nesting depth: 4 levels

Maintainability:

• Files < 500 lines
• Functions < 50 lines
• Classes < 300 lines

Documentation:

• 100% public API documented
• Docstring coverage ≥ 90%

Test Quality Metrics

Coverage:

• Overall: ≥ 80%
• Critical paths: 100%
• Core logic: ≥ 90%

Test Quality:

• No flaky tests
• Unit tests < 1 minute total
• Integration tests < 5 minutes total

Performance Benchmarks

Refer to performance-benchmarks.md for detailed criteria

Response Time:

• p50: < 50ms
• p95: < 200ms
• p99: < 500ms

Resource Usage:

• Memory: < 100MB
• CPU: < 50% single core

Automated Validation Script

The scripts/run_checks.py script automates validation:

# Run all checks
python scripts/run_checks.py --all

# Run specific checks
python scripts/run_checks.py --quality
python scripts/run_checks.py --tests
python scripts/run_checks.py --coverage
python scripts/run_checks.py --security
python scripts/run_checks.py --performance

# Generate report
python scripts/run_checks.py --all --report validation-report.md

Supporting Resources

• quality-checklist.md: Comprehensive code quality standards
• performance-benchmarks.md: Performance criteria and targets
• scripts/run_checks.py: Automated validation runner

Integration with Feature Implementation Flow

Input: Completed implementation with tests Process: Systematic validation against all criteria Output: Validation report + approval for PR Next Step: Create pull request or deploy

Validation Checklist Summary

Quality ✓

• Code formatted (Black)
• Type checked (mypy)
• Linted (no errors/warnings)
• Files < 500 lines
• Functions documented
• Quality checklist complete

Testing ✓

• All tests passing
• Coverage ≥ 80%
• Core logic ≥ 90% coverage
• No flaky tests
• Tests run quickly

Performance ✓

• Response time < target
• Throughput meets requirements
• Memory usage reasonable
• No performance regressions

Security ✓

• Input validation complete
• No hardcoded secrets
• Dependencies scanned
• Security checklist complete

Requirements ✓

• Acceptance criteria met
• User stories fulfilled
• Edge cases handled
• No regressions

Documentation ✓

• Code documented
• Technical docs complete
• User docs (if applicable)
• CHANGELOG updated

Integration ✓

• Integration tests passing
• No breaking changes
• Backward compatible

Final Approval ✓

• All checklists complete
• Validation report generated
• Ready for pull request
• Stakeholder approval (if required)

Sign-off

Feature: [Feature Name] Validated By: [Your Name] Date: [YYYY-MM-DD]

Status: ☐ Approved ☐ Needs Work

Notes: [Any additional notes or concerns]

What to Do If Validation Fails

Quality Issues:

1. Fix formatting: black src/ tests/
2. Fix type errors: Review mypy output
3. Fix lint errors: Review flake8 output
4. Refactor large files/functions

Coverage Issues:

1. Identify untested code: pytest --cov-report=html
2. Add missing tests
3. Review edge cases
4. Add error condition tests

Performance Issues:

1. Profile code: python -m cProfile
2. Optimize hot paths
3. Add caching where appropriate
4. Optimize database queries

Security Issues:

1. Address vulnerabilities: pip-audit
2. Review input validation
3. Check secrets management
4. Run security checklist again

Requirement Issues:

1. Review acceptance criteria
2. Implement missing functionality
3. Test edge cases
4. Verify with stakeholders

After Fixes:

• Re-run validation
• Update validation report
• Verify all checks pass
• Proceed to PR