VAMFI Regression Plan
Define the scope, priority, and automation approach for regression testing. Identify flaky tests and calculate automation ROI to ensure regression testing is sustainable and effective.
Overview
Regression testing answers: "Does the new change break anything that was working before?" A regression plan prevents over-testing (running everything every time) and under-testing (missing critical paths).
Regression Planning Process
Step 1: Define Regression Scope
For each release or change, determine which test suites must run:
| Trigger | Regression Scope | Rationale |
|---|---|---|
| Any PR merge | Unit + lint | Fast feedback, catches most errors |
| Staging deploy | Unit + integration + E2E smoke | Validates environment |
| Production release | Full suite + performance baseline | Maximum safety |
| Database schema change | Data migration tests + full integration | Schema regressions are high risk |
| Authentication change | Security tests + full E2E | Auth breakage is critical |
| Third-party update | Integration tests for that provider | Contract validation |
Step 2: Risk-Based Prioritisation
Score each test suite by risk:
| Test Suite | Business Impact | Change Frequency | Automation | Priority |
|---|---|---|---|---|
| [Payment flow E2E] | Critical | Low | 100% | P0 — always run |
| [User registration] | High | Medium | 90% | P0 — always run |
| [Report generation] | Medium | Low | 70% | P1 — run on staging |
| [Admin settings] | Low | Low | 50% | P2 — run on release |
Step 3: Flaky Test Identification
Identify flaky tests (tests that pass and fail without code changes):
| Test ID | Failure Rate | Root Cause Category | Recommendation |
|---|---|---|---|
| [TC-042] | 15% | Timing/async issue | Fix or quarantine |
| [TC-103] | 30% | Test data collision | Isolate test data |
| [TC-201] | 8% | External service dependency | Mock the service |
Root cause categories:
- Timing: test relies on async operations without proper awaiting
- Test data: shared test data mutated by concurrent tests
- External dependency: real API/service called, subject to availability
- Order dependency: test relies on side effects from another test
- Environment: test behaves differently across environments
Step 4: Automation ROI Assessment
For each manual regression area, calculate automation ROI:
Manual effort per release: [time] hours × [£/hr] = [£cost]
Automation build cost: [time] hours × [£/hr] = [£build cost]
Maintenance per release: [time] hours × [£/hr] = [£maintenance]
Annual saving: (manual cost - maintenance) × releases/year
Payback period: build cost / annual saving
ROI: (annual saving - build cost) / build cost × 100%
Recommend automation where:
- Payback period < 6 months, OR
- Risk is Critical/High and manual effort is > 30 minutes
Step 5: Regression Execution Plan
| Phase | Suite | Owner | Duration | Pass Criteria |
|---|---|---|---|---|
| PR gate | Unit + lint | CI/CD | < 5 min | 100% pass |
| Staging | Integration + smoke | CI/CD | < 15 min | 100% pass |
| Release | Full suite | QA Lead | < 45 min | 100% pass, no P0 open |
Additional Resources
references/flaky-test-remediation.md— Patterns for fixing flaky tests by root cause