Test Strategy Design Skill
You are an expert in test strategy design. When the user asks you to create a test strategy, plan test coverage, design test architecture, assess testing risks, or evaluate test automation ROI, follow these detailed instructions.
Core Principles
- Risk-driven prioritization -- Test the highest-risk areas first. Risk is the product of probability of failure and business impact of failure.
- Test pyramid optimization -- Maintain the right ratio of unit, integration, and E2E tests. More fast tests at the base, fewer slow tests at the top.
- Shift-left testing -- Find defects earlier in the development cycle where they are cheaper to fix. Integrate testing into design and development phases.
- Data-informed decisions -- Use defect data, coverage reports, production incidents, and user analytics to guide testing investments.
- Environment parity -- Test environments should mirror production as closely as possible. Differences between environments are a source of escaped defects.
- Sustainable automation -- Automate tests that provide recurring value. Not every test is worth automating; some are better executed manually.
- Continuous improvement -- Measure strategy effectiveness with metrics, review quarterly, and adapt based on findings.
Project Structure
test-strategy/
assessment/
current-state-analysis.ts
risk-assessment.ts
gap-analysis.ts
maturity-model.ts
planning/
test-pyramid-calculator.ts
coverage-planner.ts
resource-allocator.ts
timeline-builder.ts
metrics/
quality-metrics.ts
automation-roi.ts
defect-analytics.ts
coverage-tracker.ts
templates/
strategy-document.md
test-plan-template.md
risk-register.md
coverage-matrix.md
tools/
test-data-strategy.ts
environment-planner.ts
tool-selection-matrix.ts
Risk Assessment Engine
// assessment/risk-assessment.ts
export interface RiskItem {
id: string;
feature: string;
probability: number; // 1-5
impact: number; // 1-5
riskScore: number;
mitigationStrategy: string;
testingApproach: string;
automationPriority: 'critical' | 'high' | 'medium' | 'low';
}
export interface RiskMatrix {
items: RiskItem[];
overallRisk: number;
criticalPaths: string[];
recommendations: string[];
}
export function assessRisks(features: Array<{
name: string;
complexity: number;
changeFrequency: number;
userImpact: number;
hasTests: boolean;
lastIncident?: string;
}>): RiskMatrix {
const items: RiskItem[] = features.map((feature) => {
const probability = calculateProbability(feature);
const impact = feature.userImpact;
const riskScore = probability * impact;
return {
id: feature.name.toLowerCase().replace(/\s+/g, '-'),
feature: feature.name,
probability,
impact,
riskScore,
mitigationStrategy: getMitigationStrategy(riskScore),
testingApproach: getTestingApproach(riskScore, feature),
automationPriority: getAutomationPriority(riskScore),
};
});
const sorted = items.sort((a, b) => b.riskScore - a.riskScore);
const overallRisk = sorted.reduce((sum, item) => sum + item.riskScore, 0) / sorted.length;
const criticalPaths = sorted.filter((i) => i.riskScore >= 15).map((i) => i.feature);
return {
items: sorted,
overallRisk,
criticalPaths,
recommendations: generateRecommendations(sorted, overallRisk),
};
}
function calculateProbability(feature: any): number {
let score = 1;
if (feature.complexity > 3) score += 1;
if (feature.changeFrequency > 3) score += 1;
if (!feature.hasTests) score += 1;
if (feature.lastIncident) score += 1;
return Math.min(5, score);
}
function getMitigationStrategy(riskScore: number): string {
if (riskScore >= 20) return 'Comprehensive automation with monitoring and alerting';
if (riskScore >= 15) return 'Full regression suite with performance testing';
if (riskScore >= 10) return 'Targeted automation for critical paths';
if (riskScore >= 5) return 'Unit tests with selective integration tests';
return 'Basic unit test coverage';
}
function getTestingApproach(riskScore: number, feature: any): string {
if (riskScore >= 15) return 'E2E + Integration + Unit + Performance + Security';
if (riskScore >= 10) return 'E2E + Integration + Unit';
if (riskScore >= 5) return 'Integration + Unit';
return 'Unit tests';
}
function getAutomationPriority(riskScore: number): RiskItem['automationPriority'] {
if (riskScore >= 20) return 'critical';
if (riskScore >= 15) return 'high';
if (riskScore >= 10) return 'medium';
return 'low';
}
function generateRecommendations(items: RiskItem[], overallRisk: number): string[] {
const recommendations: string[] = [];
const criticalCount = items.filter((i) => i.riskScore >= 15).length;
if (criticalCount > 0) {
recommendations.push(`${criticalCount} critical risk areas identified. Prioritize automation for these features.`);
}
if (overallRisk > 10) {
recommendations.push('Overall risk is high. Consider increasing QA investment and test coverage.');
}
const untested = items.filter((i) => i.testingApproach.includes('Unit') && i.riskScore > 10);
if (untested.length > 0) {
recommendations.push(`${untested.length} medium-risk features have minimal testing. Add integration tests.`);
}
return recommendations;
}
Test Pyramid Calculator
// planning/test-pyramid-calculator.ts
export interface PyramidMetrics {
unit: { count: number; percentage: number; executionTimeMs: number };
integration: { count: number; percentage: number; executionTimeMs: number };
e2e: { count: number; percentage: number; executionTimeMs: number };
ratio: string;
isHealthy: boolean;
recommendations: string[];
}
export function analyzeTestPyramid(tests: {
unit: number;
integration: number;
e2e: number;
unitTimeMs: number;
integrationTimeMs: number;
e2eTimeMs: number;
}): PyramidMetrics {
const total = tests.unit + tests.integration + tests.e2e;
const unitPct = total > 0 ? (tests.unit / total) * 100 : 0;
const integrationPct = total > 0 ? (tests.integration / total) * 100 : 0;
const e2ePct = total > 0 ? (tests.e2e / total) * 100 : 0;
const recommendations: string[] = [];
let isHealthy = true;
// Ideal pyramid: 70% unit, 20% integration, 10% E2E
if (unitPct < 50) {
recommendations.push(`Unit tests are ${unitPct.toFixed(0)}% (target: 70%). Add more unit tests.`);
isHealthy = false;
}
if (integrationPct > 40) {
recommendations.push(`Integration tests are ${integrationPct.toFixed(0)}% (target: 20%). Consider converting some to unit tests.`);
isHealthy = false;
}
if (e2ePct > 20) {
recommendations.push(`E2E tests are ${e2ePct.toFixed(0)}% (target: 10%). Too many E2E tests slow CI/CD.`);
isHealthy = false;
}
// Check execution time balance
const totalTime = tests.unitTimeMs + tests.integrationTimeMs + tests.e2eTimeMs;
if (tests.e2eTimeMs > totalTime * 0.7) {
recommendations.push('E2E tests consume >70% of execution time. Optimize or parallelize.');
}
return {
unit: { count: tests.unit, percentage: Math.round(unitPct), executionTimeMs: tests.unitTimeMs },
integration: { count: tests.integration, percentage: Math.round(integrationPct), executionTimeMs: tests.integrationTimeMs },
e2e: { count: tests.e2e, percentage: Math.round(e2ePct), executionTimeMs: tests.e2eTimeMs },
ratio: `${Math.round(unitPct)}:${Math.round(integrationPct)}:${Math.round(e2ePct)}`,
isHealthy,
recommendations,
};
}
Automation ROI Calculator
// metrics/automation-roi.ts
export interface ROIAnalysis {
totalInvestment: number;
annualSavings: number;
breakEvenMonths: number;
threeYearROI: number;
costPerExecution: { manual: number; automated: number };
recommendation: string;
}
export function calculateAutomationROI(params: {
manualExecutionTimeHours: number;
executionsPerMonth: number;
qaHourlyRate: number;
automationDevelopmentHours: number;
automationMaintenanceHoursPerMonth: number;
automatedExecutionTimeHours: number;
toolCostPerMonth: number;
sdetHourlyRate: number;
}): ROIAnalysis {
const {
manualExecutionTimeHours,
executionsPerMonth,
qaHourlyRate,
automationDevelopmentHours,
automationMaintenanceHoursPerMonth,
automatedExecutionTimeHours,
toolCostPerMonth,
sdetHourlyRate,
} = params;
// Monthly costs
const manualMonthlyCost = manualExecutionTimeHours * executionsPerMonth * qaHourlyRate;
const automatedMonthlyCost =
(automationMaintenanceHoursPerMonth * sdetHourlyRate) +
(automatedExecutionTimeHours * executionsPerMonth * sdetHourlyRate * 0.1) +
toolCostPerMonth;
// Initial investment
const totalInvestment = automationDevelopmentHours * sdetHourlyRate;
// Monthly savings
const monthlySavings = manualMonthlyCost - automatedMonthlyCost;
const annualSavings = monthlySavings * 12;
// Break-even
const breakEvenMonths = monthlySavings > 0 ? Math.ceil(totalInvestment / monthlySavings) : Infinity;
// 3-year ROI
const threeYearSavings = annualSavings * 3;
const threeYearROI = ((threeYearSavings - totalInvestment) / totalInvestment) * 100;
// Cost per execution
const manualCostPerExecution = manualExecutionTimeHours * qaHourlyRate;
const automatedCostPerExecution = automatedMonthlyCost / executionsPerMonth;
let recommendation: string;
if (breakEvenMonths <= 6) {
recommendation = 'Strong ROI. Automate immediately.';
} else if (breakEvenMonths <= 12) {
recommendation = 'Good ROI. Proceed with automation.';
} else if (breakEvenMonths <= 24) {
recommendation = 'Moderate ROI. Automate high-frequency tests first.';
} else {
recommendation = 'Low ROI. Consider partial automation or manual testing.';
}
return {
totalInvestment: Math.round(totalInvestment),
annualSavings: Math.round(annualSavings),
breakEvenMonths,
threeYearROI: Math.round(threeYearROI),
costPerExecution: {
manual: Math.round(manualCostPerExecution * 100) / 100,
automated: Math.round(automatedCostPerExecution * 100) / 100,
},
recommendation,
};
}
Coverage Planning Matrix
// planning/coverage-planner.ts
export interface CoverageTarget {
feature: string;
unitCoverage: number;
integrationCoverage: number;
e2eCoverage: number;
manualTestCases: number;
automationStatus: 'not-started' | 'in-progress' | 'complete';
estimatedEffort: number; // person-days
}
export function generateCoveragePlan(
features: Array<{
name: string;
risk: 'critical' | 'high' | 'medium' | 'low';
complexity: number;
currentCoverage: number;
}>
): CoverageTarget[] {
return features.map((feature) => {
const targets = getCoverageTargets(feature.risk);
return {
feature: feature.name,
unitCoverage: targets.unit,
integrationCoverage: targets.integration,
e2eCoverage: targets.e2e,
manualTestCases: targets.manual,
automationStatus: feature.currentCoverage > 50 ? 'in-progress' : 'not-started',
estimatedEffort: calculateEffort(feature.complexity, feature.currentCoverage, targets),
};
});
}
function getCoverageTargets(risk: string): {
unit: number; integration: number; e2e: number; manual: number;
} {
switch (risk) {
case 'critical':
return { unit: 90, integration: 80, e2e: 70, manual: 10 };
case 'high':
return { unit: 80, integration: 60, e2e: 50, manual: 5 };
case 'medium':
return { unit: 70, integration: 40, e2e: 30, manual: 3 };
case 'low':
return { unit: 50, integration: 20, e2e: 10, manual: 0 };
default:
return { unit: 60, integration: 30, e2e: 20, manual: 2 };
}
}
function calculateEffort(
complexity: number,
currentCoverage: number,
targets: { unit: number; integration: number; e2e: number }
): number {
const coverageGap = Math.max(0, targets.unit - currentCoverage);
const baseEffort = complexity * 0.5;
const coverageFactor = coverageGap / 100;
return Math.ceil(baseEffort * (1 + coverageFactor));
}
Test Environment Strategy
// tools/environment-planner.ts
export interface EnvironmentPlan {
name: string;
purpose: string;
dataStrategy: string;
refreshCadence: string;
cost: string;
ownership: string;
}
export const environmentStrategy: EnvironmentPlan[] = [
{
name: 'Local Development',
purpose: 'Developer testing with mocked dependencies',
dataStrategy: 'In-memory database with seed data',
refreshCadence: 'On every test run',
cost: 'Zero (runs locally)',
ownership: 'Individual developers',
},
{
name: 'CI Environment',
purpose: 'Automated test execution in pipeline',
dataStrategy: 'Docker-based database with migration scripts',
refreshCadence: 'Every pipeline run (ephemeral)',
cost: 'CI compute minutes',
ownership: 'DevOps team',
},
{
name: 'Staging',
purpose: 'Pre-production validation and E2E testing',
dataStrategy: 'Anonymized production data subset',
refreshCadence: 'Weekly refresh from production',
cost: 'Cloud infrastructure (reduced size)',
ownership: 'QA team',
},
{
name: 'Performance',
purpose: 'Load and performance testing',
dataStrategy: 'Production-scale synthetic data',
refreshCadence: 'Before each performance test cycle',
cost: 'Production-equivalent infrastructure',
ownership: 'Performance engineering',
},
];
Test Tool Selection Matrix
// tools/tool-selection-matrix.ts
export interface ToolEvaluation {
tool: string;
category: string;
scores: {
teamSkillFit: number; // 1-5
techStackFit: number; // 1-5
communitySupport: number; // 1-5
ciIntegration: number; // 1-5
costEfficiency: number; // 1-5
aiAgentSupport: number; // 1-5
};
totalScore: number;
recommendation: string;
}
export function evaluateTools(
tools: Array<{
tool: string;
category: string;
scores: Record<string, number>;
}>,
weights: Record<string, number> = {
teamSkillFit: 2,
techStackFit: 2,
communitySupport: 1,
ciIntegration: 1.5,
costEfficiency: 1,
aiAgentSupport: 1.5,
}
): ToolEvaluation[] {
return tools.map((t) => {
const totalScore = Object.entries(t.scores).reduce(
(sum, [key, value]) => sum + value * (weights[key] || 1),
0
);
return {
...t,
scores: t.scores as any,
totalScore: Math.round(totalScore * 100) / 100,
recommendation: totalScore > 35 ? 'Recommended' : totalScore > 25 ? 'Consider' : 'Not recommended',
};
}).sort((a, b) => b.totalScore - a.totalScore);
}
Quality Metrics Dashboard Specification
// metrics/quality-metrics.ts
export interface QualityDashboard {
overview: {
defectEscapeRate: number;
automationCoverage: number;
testPassRate: number;
meanTimeToDetect: number; // hours
meanTimeToFix: number; // hours
flakyTestPercentage: number;
};
trends: Array<{
week: string;
passRate: number;
coverage: number;
executionTime: number;
newDefects: number;
}>;
riskMap: Array<{
feature: string;
risk: number;
coverage: number;
defectCount: number;
}>;
}
export function calculateDefectEscapeRate(
defectsFoundInTesting: number,
defectsFoundInProduction: number
): number {
const total = defectsFoundInTesting + defectsFoundInProduction;
if (total === 0) return 0;
return (defectsFoundInProduction / total) * 100;
}
export function calculateAutomationROI(params: {
manualHoursPerWeek: number;
automatedHoursPerWeek: number;
automationBuildHours: number;
maintenanceHoursPerWeek: number;
qaHourlyRate: number;
}): {
weeklySavings: number;
breakEvenWeeks: number;
annualROI: number;
} {
const weeklySavings = (params.manualHoursPerWeek - params.automatedHoursPerWeek - params.maintenanceHoursPerWeek) * params.qaHourlyRate;
const totalInvestment = params.automationBuildHours * params.qaHourlyRate;
const breakEvenWeeks = weeklySavings > 0 ? Math.ceil(totalInvestment / weeklySavings) : Infinity;
const annualROI = ((weeklySavings * 52 - totalInvestment) / totalInvestment) * 100;
return { weeklySavings, breakEvenWeeks, annualROI: Math.round(annualROI) };
}
Strategy Document Template
A test strategy document should contain the following sections organized for both technical and non-technical audiences.
The Executive Summary provides a one-paragraph overview of the testing approach, key risks, and resource requirements. The Scope section defines what will and will not be tested, along with the assumptions and constraints. The Risk Assessment section lists the top risks with their probability, impact, and mitigation strategies.
The Test Approach section describes the test pyramid distribution, automation versus manual split, and which test types apply to which features. The Environment Strategy section defines each test environment, its purpose, data strategy, and ownership. The Tool Stack section lists the selected tools with justification for each choice.
The Timeline and Milestones section provides a phased plan with concrete deliverables for each phase. The Resource Requirements section specifies the team composition, skill requirements, and estimated effort. The Success Criteria section defines measurable thresholds for quality, coverage, and execution time.
The Reporting and Communication section describes how test results will be communicated, to whom, and at what frequency. The Risk Register tracks identified risks with their status and mitigation progress.
Best Practices
- Start with risk assessment -- Before writing any test, assess what can go wrong and what the impact would be. Let risk drive your testing investments.
- Document the strategy -- A test strategy should be a living document reviewed quarterly. Include scope, approach, tools, environments, and exit criteria.
- Maintain the test pyramid -- Regularly audit your test distribution. If E2E tests exceed 20% of your suite, investigate which can be pushed down to integration or unit level.
- Calculate ROI before automating -- Not every test is worth automating. Calculate the break-even point and prioritize high-frequency, high-value test cases.
- Align strategy with business goals -- Testing investments should map to business risks. Test payment flows more than about-page content.
- Plan for test data -- Test data management is often the hardest part. Design a data strategy that includes creation, isolation, and cleanup.
- Design for parallel execution -- From day one, ensure tests are independent so they can run in parallel as the suite grows.
- Include non-functional testing -- Performance, security, and accessibility testing should be part of the strategy, not afterthoughts.
- Measure and report -- Track defect escape rate, test coverage, automation coverage, and test execution time. Report trends monthly.
- Plan for maintenance -- Budget 20-30% of initial automation effort for ongoing maintenance. Tests are code and need upkeep.
Anti-Patterns
- Testing everything equally -- Not all features have equal risk. Testing low-risk features as thoroughly as high-risk ones wastes resources.
- Ice cream cone anti-pattern -- More E2E tests than unit tests creates slow, brittle, expensive test suites. Invert the pyramid.
- No strategy document -- Testing without a documented strategy leads to ad-hoc, inconsistent coverage with gaps in critical areas.
- Ignoring maintenance costs -- Automating 1000 tests without budgeting for maintenance leads to a suite of broken, ignored tests.
- Copy-pasting strategies -- Every project has unique risks. Using a generic strategy template without customization misses critical context.
- Not involving developers -- Test strategy is a team responsibility. Strategies created by QA in isolation miss developer perspectives on risk and testability.
- Measuring only coverage percentage -- 80% code coverage with weak assertions provides false confidence. Measure mutation testing scores too.
- Not reviewing production incidents -- Every production incident should feed back into the test strategy. If a bug escaped, the strategy has a gap.
- Over-automating UI tests -- UI tests are expensive to maintain. Automate business-critical paths and use exploratory testing for the rest.
- Ignoring test environment costs -- Test environments have infrastructure costs. Plan for environments as part of the testing budget.
1---2name: test-strategy-design3description: Comprehensive test strategy design methodology covering risk-based testing, test pyramid optimization, coverage planning, environment strategy, data management, and test automation ROI analysis for enterprise and startup contexts.4version: 1.0.05author: thetestingacademy6license: MIT7tags: [test-strategy, risk-based-testing, test-pyramid, coverage-planning, test-planning, test-management, qa-strategy, test-architecture, quality-strategy]8testingTypes: [unit, integration, e2e, api, performance, security, accessibility]9frameworks: [playwright, vitest, jest, cypress, selenium]10languages: [typescript, javascript, python, java]11domains: [web, backend, api, mobile, enterprise]12agents: [claude-code, cursor, github-copilot, windsurf, codex, aider, continue, cline, zed, bolt, gemini-cli, amp]13metadata:14 source: https://skillmd.com/skills/pramoddutta/test-strategy-design15---1617# Test Strategy Design Skill1819You are an expert in test strategy design. When the user asks you to create a test strategy, plan test coverage, design test architecture, assess testing risks, or evaluate test automation ROI, follow these detailed instructions.2021## Core Principles22231. **Risk-driven prioritization** -- Test the highest-risk areas first. Risk is the product of probability of failure and business impact of failure.242. **Test pyramid optimization** -- Maintain the right ratio of unit, integration, and E2E tests. More fast tests at the base, fewer slow tests at the top.253. **Shift-left testing** -- Find defects earlier in the development cycle where they are cheaper to fix. Integrate testing into design and development phases.264. **Data-informed decisions** -- Use defect data, coverage reports, production incidents, and user analytics to guide testing investments.275. **Environment parity** -- Test environments should mirror production as closely as possible. Differences between environments are a source of escaped defects.286. **Sustainable automation** -- Automate tests that provide recurring value. Not every test is worth automating; some are better executed manually.297. **Continuous improvement** -- Measure strategy effectiveness with metrics, review quarterly, and adapt based on findings.3031## Project Structure3233```34test-strategy/35 assessment/36 current-state-analysis.ts37 risk-assessment.ts38 gap-analysis.ts39 maturity-model.ts40 planning/41 test-pyramid-calculator.ts42 coverage-planner.ts43 resource-allocator.ts44 timeline-builder.ts45 metrics/46 quality-metrics.ts47 automation-roi.ts48 defect-analytics.ts49 coverage-tracker.ts50 templates/51 strategy-document.md52 test-plan-template.md53 risk-register.md54 coverage-matrix.md55 tools/56 test-data-strategy.ts57 environment-planner.ts58 tool-selection-matrix.ts59```6061## Risk Assessment Engine6263```typescript64// assessment/risk-assessment.ts65export interface RiskItem {66 id: string;67 feature: string;68 probability: number; // 1-569 impact: number; // 1-570 riskScore: number;71 mitigationStrategy: string;72 testingApproach: string;73 automationPriority: 'critical' | 'high' | 'medium' | 'low';74}7576export interface RiskMatrix {77 items: RiskItem[];78 overallRisk: number;79 criticalPaths: string[];80 recommendations: string[];81}8283export function assessRisks(features: Array<{84 name: string;85 complexity: number;86 changeFrequency: number;87 userImpact: number;88 hasTests: boolean;89 lastIncident?: string;90}>): RiskMatrix {91 const items: RiskItem[] = features.map((feature) => {92 const probability = calculateProbability(feature);93 const impact = feature.userImpact;94 const riskScore = probability * impact;9596 return {97 id: feature.name.toLowerCase().replace(/\s+/g, '-'),98 feature: feature.name,99 probability,100 impact,101 riskScore,102 mitigationStrategy: getMitigationStrategy(riskScore),103 testingApproach: getTestingApproach(riskScore, feature),104 automationPriority: getAutomationPriority(riskScore),105 };106 });107108 const sorted = items.sort((a, b) => b.riskScore - a.riskScore);109 const overallRisk = sorted.reduce((sum, item) => sum + item.riskScore, 0) / sorted.length;110 const criticalPaths = sorted.filter((i) => i.riskScore >= 15).map((i) => i.feature);111112 return {113 items: sorted,114 overallRisk,115 criticalPaths,116 recommendations: generateRecommendations(sorted, overallRisk),117 };118}119120function calculateProbability(feature: any): number {121 let score = 1;122 if (feature.complexity > 3) score += 1;123 if (feature.changeFrequency > 3) score += 1;124 if (!feature.hasTests) score += 1;125 if (feature.lastIncident) score += 1;126 return Math.min(5, score);127}128129function getMitigationStrategy(riskScore: number): string {130 if (riskScore >= 20) return 'Comprehensive automation with monitoring and alerting';131 if (riskScore >= 15) return 'Full regression suite with performance testing';132 if (riskScore >= 10) return 'Targeted automation for critical paths';133 if (riskScore >= 5) return 'Unit tests with selective integration tests';134 return 'Basic unit test coverage';135}136137function getTestingApproach(riskScore: number, feature: any): string {138 if (riskScore >= 15) return 'E2E + Integration + Unit + Performance + Security';139 if (riskScore >= 10) return 'E2E + Integration + Unit';140 if (riskScore >= 5) return 'Integration + Unit';141 return 'Unit tests';142}143144function getAutomationPriority(riskScore: number): RiskItem['automationPriority'] {145 if (riskScore >= 20) return 'critical';146 if (riskScore >= 15) return 'high';147 if (riskScore >= 10) return 'medium';148 return 'low';149}150151function generateRecommendations(items: RiskItem[], overallRisk: number): string[] {152 const recommendations: string[] = [];153 const criticalCount = items.filter((i) => i.riskScore >= 15).length;154155 if (criticalCount > 0) {156 recommendations.push(`${criticalCount} critical risk areas identified. Prioritize automation for these features.`);157 }158 if (overallRisk > 10) {159 recommendations.push('Overall risk is high. Consider increasing QA investment and test coverage.');160 }161162 const untested = items.filter((i) => i.testingApproach.includes('Unit') && i.riskScore > 10);163 if (untested.length > 0) {164 recommendations.push(`${untested.length} medium-risk features have minimal testing. Add integration tests.`);165 }166167 return recommendations;168}169```170171## Test Pyramid Calculator172173```typescript174// planning/test-pyramid-calculator.ts175export interface PyramidMetrics {176 unit: { count: number; percentage: number; executionTimeMs: number };177 integration: { count: number; percentage: number; executionTimeMs: number };178 e2e: { count: number; percentage: number; executionTimeMs: number };179 ratio: string;180 isHealthy: boolean;181 recommendations: string[];182}183184export function analyzeTestPyramid(tests: {185 unit: number;186 integration: number;187 e2e: number;188 unitTimeMs: number;189 integrationTimeMs: number;190 e2eTimeMs: number;191}): PyramidMetrics {192 const total = tests.unit + tests.integration + tests.e2e;193 const unitPct = total > 0 ? (tests.unit / total) * 100 : 0;194 const integrationPct = total > 0 ? (tests.integration / total) * 100 : 0;195 const e2ePct = total > 0 ? (tests.e2e / total) * 100 : 0;196197 const recommendations: string[] = [];198 let isHealthy = true;199200 // Ideal pyramid: 70% unit, 20% integration, 10% E2E201 if (unitPct < 50) {202 recommendations.push(`Unit tests are ${unitPct.toFixed(0)}% (target: 70%). Add more unit tests.`);203 isHealthy = false;204 }205 if (integrationPct > 40) {206 recommendations.push(`Integration tests are ${integrationPct.toFixed(0)}% (target: 20%). Consider converting some to unit tests.`);207 isHealthy = false;208 }209 if (e2ePct > 20) {210 recommendations.push(`E2E tests are ${e2ePct.toFixed(0)}% (target: 10%). Too many E2E tests slow CI/CD.`);211 isHealthy = false;212 }213214 // Check execution time balance215 const totalTime = tests.unitTimeMs + tests.integrationTimeMs + tests.e2eTimeMs;216 if (tests.e2eTimeMs > totalTime * 0.7) {217 recommendations.push('E2E tests consume >70% of execution time. Optimize or parallelize.');218 }219220 return {221 unit: { count: tests.unit, percentage: Math.round(unitPct), executionTimeMs: tests.unitTimeMs },222 integration: { count: tests.integration, percentage: Math.round(integrationPct), executionTimeMs: tests.integrationTimeMs },223 e2e: { count: tests.e2e, percentage: Math.round(e2ePct), executionTimeMs: tests.e2eTimeMs },224 ratio: `${Math.round(unitPct)}:${Math.round(integrationPct)}:${Math.round(e2ePct)}`,225 isHealthy,226 recommendations,227 };228}229```230231## Automation ROI Calculator232233```typescript234// metrics/automation-roi.ts235export interface ROIAnalysis {236 totalInvestment: number;237 annualSavings: number;238 breakEvenMonths: number;239 threeYearROI: number;240 costPerExecution: { manual: number; automated: number };241 recommendation: string;242}243244export function calculateAutomationROI(params: {245 manualExecutionTimeHours: number;246 executionsPerMonth: number;247 qaHourlyRate: number;248 automationDevelopmentHours: number;249 automationMaintenanceHoursPerMonth: number;250 automatedExecutionTimeHours: number;251 toolCostPerMonth: number;252 sdetHourlyRate: number;253}): ROIAnalysis {254 const {255 manualExecutionTimeHours,256 executionsPerMonth,257 qaHourlyRate,258 automationDevelopmentHours,259 automationMaintenanceHoursPerMonth,260 automatedExecutionTimeHours,261 toolCostPerMonth,262 sdetHourlyRate,263 } = params;264265 // Monthly costs266 const manualMonthlyCost = manualExecutionTimeHours * executionsPerMonth * qaHourlyRate;267 const automatedMonthlyCost =268 (automationMaintenanceHoursPerMonth * sdetHourlyRate) +269 (automatedExecutionTimeHours * executionsPerMonth * sdetHourlyRate * 0.1) +270 toolCostPerMonth;271272 // Initial investment273 const totalInvestment = automationDevelopmentHours * sdetHourlyRate;274275 // Monthly savings276 const monthlySavings = manualMonthlyCost - automatedMonthlyCost;277 const annualSavings = monthlySavings * 12;278279 // Break-even280 const breakEvenMonths = monthlySavings > 0 ? Math.ceil(totalInvestment / monthlySavings) : Infinity;281282 // 3-year ROI283 const threeYearSavings = annualSavings * 3;284 const threeYearROI = ((threeYearSavings - totalInvestment) / totalInvestment) * 100;285286 // Cost per execution287 const manualCostPerExecution = manualExecutionTimeHours * qaHourlyRate;288 const automatedCostPerExecution = automatedMonthlyCost / executionsPerMonth;289290 let recommendation: string;291 if (breakEvenMonths <= 6) {292 recommendation = 'Strong ROI. Automate immediately.';293 } else if (breakEvenMonths <= 12) {294 recommendation = 'Good ROI. Proceed with automation.';295 } else if (breakEvenMonths <= 24) {296 recommendation = 'Moderate ROI. Automate high-frequency tests first.';297 } else {298 recommendation = 'Low ROI. Consider partial automation or manual testing.';299 }300301 return {302 totalInvestment: Math.round(totalInvestment),303 annualSavings: Math.round(annualSavings),304 breakEvenMonths,305 threeYearROI: Math.round(threeYearROI),306 costPerExecution: {307 manual: Math.round(manualCostPerExecution * 100) / 100,308 automated: Math.round(automatedCostPerExecution * 100) / 100,309 },310 recommendation,311 };312}313```314315## Coverage Planning Matrix316317```typescript318// planning/coverage-planner.ts319export interface CoverageTarget {320 feature: string;321 unitCoverage: number;322 integrationCoverage: number;323 e2eCoverage: number;324 manualTestCases: number;325 automationStatus: 'not-started' | 'in-progress' | 'complete';326 estimatedEffort: number; // person-days327}328329export function generateCoveragePlan(330 features: Array<{331 name: string;332 risk: 'critical' | 'high' | 'medium' | 'low';333 complexity: number;334 currentCoverage: number;335 }>336): CoverageTarget[] {337 return features.map((feature) => {338 const targets = getCoverageTargets(feature.risk);339340 return {341 feature: feature.name,342 unitCoverage: targets.unit,343 integrationCoverage: targets.integration,344 e2eCoverage: targets.e2e,345 manualTestCases: targets.manual,346 automationStatus: feature.currentCoverage > 50 ? 'in-progress' : 'not-started',347 estimatedEffort: calculateEffort(feature.complexity, feature.currentCoverage, targets),348 };349 });350}351352function getCoverageTargets(risk: string): {353 unit: number; integration: number; e2e: number; manual: number;354} {355 switch (risk) {356 case 'critical':357 return { unit: 90, integration: 80, e2e: 70, manual: 10 };358 case 'high':359 return { unit: 80, integration: 60, e2e: 50, manual: 5 };360 case 'medium':361 return { unit: 70, integration: 40, e2e: 30, manual: 3 };362 case 'low':363 return { unit: 50, integration: 20, e2e: 10, manual: 0 };364 default:365 return { unit: 60, integration: 30, e2e: 20, manual: 2 };366 }367}368369function calculateEffort(370 complexity: number,371 currentCoverage: number,372 targets: { unit: number; integration: number; e2e: number }373): number {374 const coverageGap = Math.max(0, targets.unit - currentCoverage);375 const baseEffort = complexity * 0.5;376 const coverageFactor = coverageGap / 100;377 return Math.ceil(baseEffort * (1 + coverageFactor));378}379```380381## Test Environment Strategy382383```typescript384// tools/environment-planner.ts385export interface EnvironmentPlan {386 name: string;387 purpose: string;388 dataStrategy: string;389 refreshCadence: string;390 cost: string;391 ownership: string;392}393394export const environmentStrategy: EnvironmentPlan[] = [395 {396 name: 'Local Development',397 purpose: 'Developer testing with mocked dependencies',398 dataStrategy: 'In-memory database with seed data',399 refreshCadence: 'On every test run',400 cost: 'Zero (runs locally)',401 ownership: 'Individual developers',402 },403 {404 name: 'CI Environment',405 purpose: 'Automated test execution in pipeline',406 dataStrategy: 'Docker-based database with migration scripts',407 refreshCadence: 'Every pipeline run (ephemeral)',408 cost: 'CI compute minutes',409 ownership: 'DevOps team',410 },411 {412 name: 'Staging',413 purpose: 'Pre-production validation and E2E testing',414 dataStrategy: 'Anonymized production data subset',415 refreshCadence: 'Weekly refresh from production',416 cost: 'Cloud infrastructure (reduced size)',417 ownership: 'QA team',418 },419 {420 name: 'Performance',421 purpose: 'Load and performance testing',422 dataStrategy: 'Production-scale synthetic data',423 refreshCadence: 'Before each performance test cycle',424 cost: 'Production-equivalent infrastructure',425 ownership: 'Performance engineering',426 },427];428```429430## Test Tool Selection Matrix431432```typescript433// tools/tool-selection-matrix.ts434export interface ToolEvaluation {435 tool: string;436 category: string;437 scores: {438 teamSkillFit: number; // 1-5439 techStackFit: number; // 1-5440 communitySupport: number; // 1-5441 ciIntegration: number; // 1-5442 costEfficiency: number; // 1-5443 aiAgentSupport: number; // 1-5444 };445 totalScore: number;446 recommendation: string;447}448449export function evaluateTools(450 tools: Array<{451 tool: string;452 category: string;453 scores: Record<string, number>;454 }>,455 weights: Record<string, number> = {456 teamSkillFit: 2,457 techStackFit: 2,458 communitySupport: 1,459 ciIntegration: 1.5,460 costEfficiency: 1,461 aiAgentSupport: 1.5,462 }463): ToolEvaluation[] {464 return tools.map((t) => {465 const totalScore = Object.entries(t.scores).reduce(466 (sum, [key, value]) => sum + value * (weights[key] || 1),467 0468 );469470 return {471 ...t,472 scores: t.scores as any,473 totalScore: Math.round(totalScore * 100) / 100,474 recommendation: totalScore > 35 ? 'Recommended' : totalScore > 25 ? 'Consider' : 'Not recommended',475 };476 }).sort((a, b) => b.totalScore - a.totalScore);477}478```479480## Quality Metrics Dashboard Specification481482```typescript483// metrics/quality-metrics.ts484export interface QualityDashboard {485 overview: {486 defectEscapeRate: number;487 automationCoverage: number;488 testPassRate: number;489 meanTimeToDetect: number; // hours490 meanTimeToFix: number; // hours491 flakyTestPercentage: number;492 };493 trends: Array<{494 week: string;495 passRate: number;496 coverage: number;497 executionTime: number;498 newDefects: number;499 }>;500 riskMap: Array<{501 feature: string;502 risk: number;503 coverage: number;504 defectCount: number;505 }>;506}507508export function calculateDefectEscapeRate(509 defectsFoundInTesting: number,510 defectsFoundInProduction: number511): number {512 const total = defectsFoundInTesting + defectsFoundInProduction;513 if (total === 0) return 0;514 return (defectsFoundInProduction / total) * 100;515}516517export function calculateAutomationROI(params: {518 manualHoursPerWeek: number;519 automatedHoursPerWeek: number;520 automationBuildHours: number;521 maintenanceHoursPerWeek: number;522 qaHourlyRate: number;523}): {524 weeklySavings: number;525 breakEvenWeeks: number;526 annualROI: number;527} {528 const weeklySavings = (params.manualHoursPerWeek - params.automatedHoursPerWeek - params.maintenanceHoursPerWeek) * params.qaHourlyRate;529 const totalInvestment = params.automationBuildHours * params.qaHourlyRate;530 const breakEvenWeeks = weeklySavings > 0 ? Math.ceil(totalInvestment / weeklySavings) : Infinity;531 const annualROI = ((weeklySavings * 52 - totalInvestment) / totalInvestment) * 100;532533 return { weeklySavings, breakEvenWeeks, annualROI: Math.round(annualROI) };534}535```536537## Strategy Document Template538539A test strategy document should contain the following sections organized for both technical and non-technical audiences.540541The Executive Summary provides a one-paragraph overview of the testing approach, key risks, and resource requirements. The Scope section defines what will and will not be tested, along with the assumptions and constraints. The Risk Assessment section lists the top risks with their probability, impact, and mitigation strategies.542543The Test Approach section describes the test pyramid distribution, automation versus manual split, and which test types apply to which features. The Environment Strategy section defines each test environment, its purpose, data strategy, and ownership. The Tool Stack section lists the selected tools with justification for each choice.544545The Timeline and Milestones section provides a phased plan with concrete deliverables for each phase. The Resource Requirements section specifies the team composition, skill requirements, and estimated effort. The Success Criteria section defines measurable thresholds for quality, coverage, and execution time.546547The Reporting and Communication section describes how test results will be communicated, to whom, and at what frequency. The Risk Register tracks identified risks with their status and mitigation progress.548549## Best Practices5505511. **Start with risk assessment** -- Before writing any test, assess what can go wrong and what the impact would be. Let risk drive your testing investments.5522. **Document the strategy** -- A test strategy should be a living document reviewed quarterly. Include scope, approach, tools, environments, and exit criteria.5533. **Maintain the test pyramid** -- Regularly audit your test distribution. If E2E tests exceed 20% of your suite, investigate which can be pushed down to integration or unit level.5544. **Calculate ROI before automating** -- Not every test is worth automating. Calculate the break-even point and prioritize high-frequency, high-value test cases.5555. **Align strategy with business goals** -- Testing investments should map to business risks. Test payment flows more than about-page content.5566. **Plan for test data** -- Test data management is often the hardest part. Design a data strategy that includes creation, isolation, and cleanup.5577. **Design for parallel execution** -- From day one, ensure tests are independent so they can run in parallel as the suite grows.5588. **Include non-functional testing** -- Performance, security, and accessibility testing should be part of the strategy, not afterthoughts.5599. **Measure and report** -- Track defect escape rate, test coverage, automation coverage, and test execution time. Report trends monthly.56010. **Plan for maintenance** -- Budget 20-30% of initial automation effort for ongoing maintenance. Tests are code and need upkeep.561562## Anti-Patterns5635641. **Testing everything equally** -- Not all features have equal risk. Testing low-risk features as thoroughly as high-risk ones wastes resources.5652. **Ice cream cone anti-pattern** -- More E2E tests than unit tests creates slow, brittle, expensive test suites. Invert the pyramid.5663. **No strategy document** -- Testing without a documented strategy leads to ad-hoc, inconsistent coverage with gaps in critical areas.5674. **Ignoring maintenance costs** -- Automating 1000 tests without budgeting for maintenance leads to a suite of broken, ignored tests.5685. **Copy-pasting strategies** -- Every project has unique risks. Using a generic strategy template without customization misses critical context.5696. **Not involving developers** -- Test strategy is a team responsibility. Strategies created by QA in isolation miss developer perspectives on risk and testability.5707. **Measuring only coverage percentage** -- 80% code coverage with weak assertions provides false confidence. Measure mutation testing scores too.5718. **Not reviewing production incidents** -- Every production incident should feed back into the test strategy. If a bug escaped, the strategy has a gap.5729. **Over-automating UI tests** -- UI tests are expensive to maintain. Automate business-critical paths and use exploratory testing for the rest.57310. **Ignoring test environment costs** -- Test environments have infrastructure costs. Plan for environments as part of the testing budget.