Testing Automation Expert

Testing Automation Expert

Master production-grade testing strategies for Python and TypeScript applications. QA Architect level - design test ecosystems that scale.

When to Use

• Designing test strategies for features/projects
• Setting up pytest, Jest, Vitest, or Playwright
• Improving test coverage and quality
• Contract testing, accessibility, mutation testing
• CI/CD test pipeline optimization
• Test architecture review or refactoring

Triggers

• /test-plan - Design test strategy for a feature/project
• /fixture-refactor - Optimize test fixture architecture
• /e2e-setup - Set up Playwright E2E testing
• /test-audit - Review test coverage and quality
• /contract-test - Set up Pact contract testing

Reference Files

Load the appropriate reference based on need:

Test Quality Checklists

Pre-Implementation Checklist

• Test strategy defined - unit, integration, E2E, contract boundaries clear
• Fixtures planned - factory functions, DB setup, cleanup
• Mocking strategy - mock at boundaries only, not internals
• Coverage targets - minimum 80% branches/lines/functions
• CI integration - tests run on every PR with sharding if >10min
• Flakiness detection - pytest-randomly / random test order enabled

Unit Test Checklist

• Tests isolated - no shared state between tests
• One assertion focus - test one behavior per test
• Descriptive names - should_return_error_when_user_not_found
• AAA pattern - Arrange, Act, Assert clearly separated
• Edge cases covered - null, empty, boundary values
• No external dependencies - DB, network, filesystem mocked
• Property-based tests - Hypothesis/fast-check for edge case discovery

Integration Test Checklist

• Real dependencies - use Testcontainers (not SQLite for Postgres!)
• Transaction rollback - isolate tests with rollback
• API contract tested - request/response shapes verified
• Error paths tested - timeout, connection errors
• Auth flows tested - token generation, validation
• Declarative assertions - dirty-equals for complex JSON

E2E Test Checklist

• Critical paths covered - login, checkout, core flows
• Page Object Model - locators abstracted into classes
• Resilient selectors - role/label over CSS/xpath
• Network mocking - test error states without real API
• Visual regression - Docker for consistent screenshots
• Cross-browser - Chrome, Firefox, Safari, mobile
• Auth state reuse - save/load storage state
• Trace artifacts - uploaded for CI debugging

Architecture Checklist (Microservices)

• Contract tests - Pact CDCT between services
• API fuzzing - schemathesis against OpenAPI spec
• Mutation testing - mutmut/Stryker validates test quality
• Accessibility - axe-core in E2E pipeline
• Performance baselines - k6/Locust load tests
• CI sharding - parallel machines for >10min suites

Test Maintenance Checklist

• No flaky tests - deterministic, no sleep()
• Fast execution - unit <5s, integration <30s, E2E <2min
• Readable failures - clear error messages
• DRY fixtures - reusable factories, not copy-paste
• Regular cleanup - delete obsolete tests
• Quarantine flaky - .flaky.spec.ts until fixed

Testing Pyramid

         /\
        /E2E\        Few, slow, high confidence
       /-----\
      /Integ. \      Some, moderate speed
     /---------\
    / Contract  \    Service boundaries
   /-------------\
  /    Unit       \  Many, fast, isolated
 /------------------\

Test Organization

tests/
├── conftest.py / setup.ts   # Root fixtures
├── factories/               # Test data factories
│   └── user.factory.ts
├── fixtures/                # Static test data
│   └── users.json
├── unit/                    # Fast, isolated
│   ├── services/
│   └── utils/
├── integration/             # DB, external services
│   ├── api/
│   └── repositories/
├── contract/                # Pact consumer/provider
│   ├── consumer/
│   └── provider/
├── e2e/                     # Full user flows
│   ├── pages/               # Page objects
│   ├── specs/
│   └── a11y/                # Accessibility tests
└── load/                    # k6/Locust scripts
    └── api-load.js

Anti-Patterns

Specialized Testing

Contract Testing (Microservices)

# Consumer test (Python)
pact = Consumer("UserService").has_pact_with(Provider("AuthService"))

See references/specialized-testing.md for full Pact patterns.

API Fuzzing (FastAPI/OpenAPI)

schemathesis run http://localhost:8000/openapi.json

Auto-generates tests from OpenAPI spec - finds edge cases.

Mutation Testing

# Python
mutmut run && mutmut results

# TypeScript
npx stryker run

Validates test quality - surviving mutants = gaps.

Accessibility Testing

const results = await new AxeBuilder({ page }).analyze();
expect(results.violations).toEqual([]);

Quick Reference

Python (pytest)

pytest                              # Run all
pytest -m "not slow"                # Skip slow
pytest --cov=src --cov-report=html  # Coverage
pytest -n auto                      # Parallel
pytest -x --lf                      # Stop first, re-run failed
pytest -p randomly                  # Randomize order (flakiness detection)

TypeScript (Vitest)

npx vitest                  # Watch mode
npx vitest run              # Run once
npx vitest --coverage       # Coverage
npx vitest --ui             # Browser UI

Playwright

npx playwright test                 # Run all
npx playwright test --shard=1/4     # CI sharding
npx playwright test --debug         # Debug mode
npx playwright show-trace trace.zip # View trace

Recommended Packages

Python

# Core
pip install pytest pytest-asyncio pytest-cov pytest-xdist
pip install pytest-mock pytest-randomly time-machine

# Advanced
pip install syrupy hypothesis dirty-equals
pip install testcontainers[postgres] httpx
pip install schemathesis pact-python mutmut
pip install playwright pytest-playwright axe-playwright-python
pip install locust  # Load testing

TypeScript

# Vitest (recommended for new projects)
npm install -D vitest @vitest/coverage-v8 @vitest/ui

# Jest (legacy/React Native)
npm install -D jest ts-jest @types/jest

# Shared
npm install -D supertest @types/supertest @faker-js/faker
npm install -D @testcontainers/postgresql

# Playwright
npm init playwright@latest
npm install -D @axe-core/playwright

# Contract & Mutation
npm install -D @pact-foundation/pact
npm install -D @stryker-mutator/core @stryker-mutator/vitest-runner