qa-aman Overview
Based on "Accelerate" by Nicole Forsgren, Jez Humble & Gene Kim. The DORA research (DevOps Research and Assessment) identified four metrics that predict both software delivery performance and organizational outcomes. High-performing teams excel on all four. QA directly influences two of them - change failure rate and mean time to restore - and enables the other two through fast, reliable test automation.
QA Metrics
Measure, analyze, and communicate quality outcomes through data.
When to Use
- Building a quality dashboard for a sprint review or exec report
- Calculating defect density or escape rate for a release
- Analyzing defect trends over time to surface systemic issues
- Producing a weekly or monthly QA health report
- Demonstrating the value of QA investment to leadership
Core Metrics
DORA Metrics (Accelerate Framework)
The four metrics that differentiate elite from low-performing engineering teams:
| Metric | What it measures | Elite benchmark | QA influence |
|---|---|---|---|
| Deployment Frequency | How often code ships to production | On-demand (multiple/day) | Fast, reliable automation enables frequent deploys |
| Lead Time for Changes | Commit to production time | < 1 hour | Slow QA cycles lengthen lead time |
| Change Failure Rate | % of releases causing production incidents | 0-15% | Test coverage directly reduces this |
| Mean Time to Restore (MTTR) | Time to recover from production failure | < 1 hour | Good observability + runbooks reduce this |
QA's accountability: Own change failure rate and contribute to lead time. If change failure rate is above 15%, testing strategy needs a root cause review. If QA is a bottleneck in lead time, automate or parallelize.
Defect Metrics
Defect Density Number of defects per unit of functionality.
- Formula: Total defects / feature area (story points, function points, or KLOC)
- Use: compare quality across sprints or modules
- Target: trending down over time in stable areas
Defect Escape Rate (Leakage) Percentage of defects found by customers (production) vs QA (pre-release).
- Formula: (Production defects / Total defects) × 100
- Use: measures QA effectiveness at catching defects before they ship
- Target: < 10% escape rate is strong. > 20% signals a coverage problem.
Defect Detection Efficiency (DDE) How early defects are caught in the development lifecycle.
- Formula: (Defects found pre-release / Total defects including production) × 100
- Use: rewards early-stage testing; earlier detection = cheaper fix
- Target: > 90%
Defect Aging Average time from defect open to resolution.
- Formula: Sum of (close date - open date) / number of defects closed
- Use: identifies bottlenecks in the fix cycle
- Watch for: P1/P2 defects staying open > 48 hours
Test Execution Metrics
Test Pass Rate Percentage of test cases that passed in a given cycle.
- Formula: (Passed cases / Total executed) × 100
- Caveat: pass rate is only meaningful if test cases are well-designed. High pass rate + high escape rate = weak test suite.
Test Coverage Percentage of requirements or features covered by test cases.
- Formula: (Features with test cases / Total features) × 100
- Track separately: unit, integration, E2E coverage
Automation Coverage Percentage of regression cases automated.
- Formula: (Automated cases / Total regression cases) × 100
- Target: 70%+ automation in stable product areas
Test Execution Velocity Test cases executed per day or per sprint.
- Use: capacity planning, identifying execution bottlenecks
Release Quality Metrics
Production Defect Rate Defects per release found in production within 30 days of ship.
- Use: lagging indicator of release quality
- Track by severity: P1/P2 in production is a critical signal
Mean Time to Detect (MTTD) Average time between a defect being introduced and being found.
- Shorter MTTD = earlier detection = cheaper fix
Regression Rate Percentage of releases where a previously passing feature breaks.
- Formula: (Releases with regressions / Total releases) × 100
- Use: measures stability of the regression suite
Workflow
Step 1: Identify What to Measure
Not every metric is useful for every context. Match metrics to questions:
| Question | Metrics to Use |
|---|---|
| Is quality improving over time? | Defect density trend, escape rate trend |
| Is QA catching bugs early enough? | DDE, MTTD, defect by phase |
| Are we shipping stable releases? | Regression rate, production defect rate |
| Is automation keeping up? | Automation coverage, execution velocity |
| Where are the quality hotspots? | Defect density by module/team |
Step 2: Collect Data
Define your data sources:
- Defect tracker (Jira, Linear, GitHub Issues): open/closed defects, severity, discovery phase, resolution time
- Test management (TestRail, Zephyr, spreadsheet): executed cases, pass/fail, coverage
- CI/CD pipeline: test run results, flaky test rate, build failure rate
- Production monitoring (Datadog, PagerDuty): production incidents, customer-reported bugs
Step 3: Calculate and Visualize
For a sprint report, calculate at minimum:
- Defects opened vs closed (net trend)
- Escape rate for the sprint
- Regression pass rate
- Any P1/P2 defects in production
For a quarterly report, add:
- Defect density by module (hotspot map)
- Escape rate trend (12-week rolling)
- Automation coverage change
- DDE trend
Step 4: Write the Quality Report
Sprint Quality Report template:
Sprint [n] Quality Report - [dates]
Team: [team name]
QA Lead: [name]
Test Execution:
Cases executed: [n]
Pass rate: [x]%
Automation coverage: [x]%
Defects:
Opened: [n] (P1: [n] | P2: [n] | P3: [n])
Closed: [n]
Escaped to production: [n]
Escape rate: [x]%
Release Quality:
Production defects (this release): [n]
Regression rate: PASS / FAIL
Sign-off: GO / NO-GO
Hotspots:
[area with disproportionate defects and why]
Trend:
[1-2 sentences: is quality improving, stable, or degrading?]
Actions:
[specific action, owner, due date]
Quarterly Quality Dashboard sections:
- Headline scorecard (4-5 KPIs with trend arrows)
- Defect trend chart (12-week rolling, by severity)
- Escape rate chart (by release)
- Automation coverage progress
- Module hotspot map (defect density by area)
- Key wins and risks
- Actions for next quarter
Step 5: Drive Action
Metrics without action are reporting theater. For every negative trend:
- State the root cause hypothesis
- Propose one specific action
- Assign an owner and a deadline
- Track it to resolution in the next report
Anti-Patterns
Vanity metrics "We ran 500 test cases" is not a quality metric. It is a volume metric. Pass rate, escape rate, and defect density are quality metrics. Distinguish between activity (what QA did) and outcomes (what quality resulted).
Reporting in isolation A 95% pass rate means nothing without context. Is that up or down from last sprint? Is the test suite getting weaker or stronger? Always show trend, not just point-in-time.
No action items A quality report with no action items is a postcard. If metrics show a problem, someone must own fixing it. If metrics show everything is fine, note what is working and why.
Measuring only pre-release defects Escape rate requires production data. QA teams that only track what they found - not what customers found - are grading their own homework. Include production defect rate in every quarterly review.
Automation coverage without flaky test tracking An automation suite that is 80% automated but 30% flaky is worse than 60% automated with stable tests. Track flaky test rate alongside automation coverage. Flaky tests erode confidence faster than low coverage.
Quality Checklist
Before publishing any quality report:
- Metrics are calculated from actual data, not estimates
- Escape rate includes production defects (not just pre-release)
- Trend context provided (not just point-in-time numbers)
- Every negative metric has a root cause hypothesis
- Every action item has an owner and a due date
- Automation coverage and flaky test rate reported together
- Report is versioned and stored where stakeholders can access it
- P1/P2 production defects called out by name, not buried in totals