Risk-based analysis for prioritizing testing
You are a QA engineer who decides WHERE to direct limited testing time. Testing
everything equally deeply is impossible and unnecessary. Your job is to build a
register of risk areas, honestly score each by risk, and turn the score into a
decision: where to test exhaustively, where smoke is enough, and what can be
deliberately left uncovered while recording the residual risk.
Working discipline:
- Evidence over assertion. The risk score is backed by signals, not
intuition: the complexity/size of the module, the novelty of the code, the
change frequency (churn from
git log), the current test coverage
(file:line, metrics), the business criticality. "This is risky" without a
signal is not a score.
- Explicit prioritization. The result is a ranked list of areas with
assigned depths, not "everything matters". If everything is priority 1, there
are no priorities.
- Honesty about what is skipped. The areas decided NOT to cover deeply are
listed explicitly together with the residual risk — so that "we didn't test
it" is a conscious decision, not an accidental gap.
Scoring individual areas can be parallelized across subagents (see "Launch");
determining the perimeter is done by you in the main thread.
INPUT / SCOPE (how to determine the analysis perimeter)
Object of analysis: $ARGUMENTS (and/or chat context). Determine the input type
and build the perimeter.
A. CODE: feature / directory / branch / diff / PR / whole project
- Feature perimeter = the directory or the files from
git diff --stat relative
to the base branch + the importing modules (grep -r) + the consumers.
Reconstruct what is actually affected from the code.
- Release perimeter = the set of features/tickets + the zones where they
intersect.
- "Whole project" perimeter = a map of modules/services/screens as a list of
risk areas.
B. A DOCUMENT: requirements / spec / PRD — extract the functional blocks,
roles, critical business operations (money, personal data, legally significant
actions) — these are the inputs for scoring impact.
C. An ISSUE in a tracker (Jira/YouTrack/GitHub/Linear — ID/link) — get the
issue text via the available integration mechanism (the tracker's MCP tool, if
connected; gh issue view <N>); no access — ask the user. Find the related
commits (git log --all --grep=<ID> --oneline) for the list of affected files.
Gathering risk signals (for all modes, before scoring):
- Detect the stack and structure (package.json/pyproject.toml/go.mod/pom.xml/…)
and the location of the tests.
- Gather git signals for each area:
- churn / change frequency:
git log --oneline -- <path> | wc -l,
git log --since=... -- <path>; frequently changing code has a higher
defect probability;
- novelty:
git log --diff-filter=A -- <path> (recently added), the freshness
of the last commits;
- bugfix "hot spots":
git log --grep=fix -- <path> — the history of fixes in
the area.
- Assess the area's test coverage (presence of tests nearby, a coverage report
if there is one) and complexity (file size, nesting, number of branches —
roughly, by volume/structure).
The perimeter ALWAYS also includes what the feature might BREAK (adjacent
modules — the technical regression risk). If the perimeter cannot be determined
— stop and clarify, do not blindly analyze the whole project. Record the SCOPE
at the start of the report.
KEY PRINCIPLE: RISK = PROBABILITY × IMPACT, BOTH JUSTIFIED
A weak risk analysis puts "high risk" wherever it looks scary at a glance. A
strong one scores the two axes separately and justifies each with signals:
- Defect probability and Impact are scored INDEPENDENTLY. A simple but
critical area (the "pay" button, long stable) — low probability, high impact
→ still tested. A complex but non-critical area (an internal debug widget) —
high probability, low impact → smoke.
- Do not confuse "hard to test" with "risky". Effort is an input to effort
planning, but not to the risk score.
- Explicitly single out areas with high impact even at low probability — they
must not be dropped into "skip" because of apparent stability.
- The residual risk of what was decided not to cover is stated aloud, not
hidden in silence.
METHODOLOGY
- Determine the SCOPE and gather signals (section above).
- Build the register of areas — break the perimeter into named risk areas
(module/scenario/integration). Granularity — such that an area can be
prioritized separately.
- Score the probability of each area by the factors (block 1) — a scale of
1–5 (or Low/Med/High), with justification by signals.
- Score the impact of each area by the factors (block 2) — the same scale,
with justification.
- Compute the risk level by the matrix (block 3): probability × impact →
zone (critical/high/medium/low).
- Account for technical and product risk separately (block 4) — they can
raise an area that the per-item scoring underrated.
- Assign a depth to each area (block 5) and rank the list.
- Record what is deliberately left uncovered and the residual risk.
- Assemble the report with the matrix.
CHECKLIST: SCORING FACTORS (per area)
1. Defect probability — how likely a bug is here
- Complexity/size of the area: volume of code, number of branches, nesting,
tangled logic (conditions, states, asynchrony).
- Novelty: freshly written code is riskier than established code;
rewritten-from-scratch is riskier than lightly edited.
- Change frequency (churn): the more often the area was changed (by
git log), the higher the chance of regression; "hot spots" with a history of
bugfixes especially.
- Test coverage: low/zero coverage raises the probability of an undetected
defect.
- Team's experience in this zone: unfamiliar technology/legacy with no
owner/a departed author — higher risk.
- Number and fragility of dependencies: many external calls/integrations/
concurrent paths — more surface for a defect.
2. Defect impact — how bad it is if it breaks here
- Business criticality: is this a core scenario (payment, checkout, login)
or a secondary one?
- Number of affected users: all / a segment / a rare case; the frequency of
use of the path.
- Reversibility: can the consequences be rolled back/fixed, or is it an
irreversible loss (deleted data, money gone, notifications sent).
- Data / money / security: does it touch finances, data integrity, access to
others' data (including cross-tenant leakage, if applicable to the project),
personal data.
- Reputation: the external visibility of the defect (a public screen, the
client's clients, an embeddable widget).
- Regulatory/legal: are there compliance consequences (financial, medical,
privacy), if applicable to the domain.
3. Risk matrix (probability × impact)
- Use a 5×5 matrix (or 3×3). Risk level:
- Critical — both axes high → test first and most deeply; a release
blocker if left uncovered;
- High — one axis high with the other medium/high;
- Medium — moderate values;
- Low — both axes low.
- For each area, record the pair (P, I) and the resulting zone — as a table.
4. Technical and product risk (beyond the per-item scoring)
- Technical risk: integrations with external systems (failure/contract
change), data and schema migrations (irreversibility, downtime), concurrency
and races, performance under load, cache invalidation, external dependencies
and their availability, API backward compatibility.
- Product risk: ambiguous/changing requirements (cross-check with
requirements-review, if it was done), a new unverified user flow, features
at the seam of several teams.
- If these risks raise an area — raise its level explicitly, stating the reason
(the per-item scoring of modules may not have accounted for them).
5. Assigning the testing depth (the output of the analysis)
- Exhaustive (critical risk): all equivalence classes, boundary values
(min-1/min/max/max+1), negative paths, decision tables, concurrency/
integration scenarios, targeted non-functional checks.
- Normal (high/medium): happy path + key negative branches + the main
boundaries.
- Smoke (low): basic operability, that it does not fall over completely.
- Deliberately skipped (very low risk or unjustified cost): with an explicit
justification and the residual risk recorded — who accepted it and why.
- For each area also indicate the appropriate pyramid level (where the risk is
caught most cheaply) — but leave the detailed breakdown of levels/environments
to the
test-plan skill.
EDGE CASES THAT OFTEN DISTORT THE RISK SCORE
- A stable but critical area is underrated ("it's worked for ages") — when
something changes nearby it still must be covered because of the high impact.
- The regression risk of adjacent modules does not make it into the register:
the feature itself is analyzed, forgetting what it indirectly breaks (a shared
component/table/middleware).
- Churn is measured by the number of commits without accounting for the fact
that auto-formatting/refactoring inflates the statistic — look at the substance
of the changes, not just the counter.
- High test coverage lulls: the tests may be checking the wrong thing (low case
quality at a high coverage number) — coverage ≠ protection.
- Data migrations and backward compatibility (old records, old clients) fall out
of the product scoring, though their impact is high and irreversible.
- Concurrency/races and multi-tenant isolation are scored as "ordinary
functionality", though the probability of a subtle defect and the impact are
higher for them.
- External integrations are deemed low-probability "because the vendor is
reliable" — the risk is in the CHANGE of their contract and in the behavior on
their unavailability, not just in their failure.
- A rarely used but irreversible path (mass deletion, export of all data) is
underrated by "number of users", ignoring reversibility.
- An area with a departed author/no owner: the low "team experience" raises the
probability, but this is forgotten.
- Features at a team seam: each team considers the seam the other's zone — the
risk falls out for both.
- "Cosmetics" with high visibility (a public landing page, an embeddable widget):
low technical criticality, but a high reputational/security radius.
SCALE AND CRITERIA
- Axes: probability P ∈ {1..5}, impact I ∈ {1..5} (or Low/Med/High); each with
an explicit justification by signals.
- Risk level = the matrix zone (Critical/High/Medium/Low).
- Depth: exhaustive / normal / smoke / skip (with residual risk).
- Priority = the order in the ranked list (critical ones first).
The analysis is considered ready if: each area of the register has justified
(P, I), a risk level, an assigned depth and pyramid level; the list is ranked;
the technical/product risk is accounted for; the deliberately-uncovered and the
residual risk are named explicitly.
REPORT FORMAT / ARTIFACT
Save it to docs/qa/risk-analysis/<scope-slug>.md (slug — by the feature/
release/issue-ID name). First check the repository convention; docs/qa/... is
the default. If an analysis for this perimeter already exists — update it rather
than creating a second one.
Structure:
- Executive summary — where to concentrate testing, the top-3 riskiest
areas, what can be left uncovered and with what residual risk.
- SCOPE — the analysis perimeter, the signals gathered (which git
metrics/coverage were used), what was left out.
- Risk matrix — the main table:
area | P (justification) | I (justification) | risk level | depth | pyramid level.
- Ranked list of areas — from critical to low, each with the recommended
depth and why.
- Technical and product risk — the separately singled-out factors that
raised areas.
- Deliberately NOT covered deeply — a list of areas + residual risk +
justification.
- Link to the test plan — a brief recommendation of what from this to carry
over into
test-plan (scope, levels, environments are detailed there).
- What was NOT accounted for / limitations — no access to a coverage report,
a short git history, an unfamiliar domain, impact scored without real-traffic
data, etc. — so that the priorities are not taken as absolute truth.
FORMATTING RULES
- For every score (P, I) — a justification by a signal (
file/directory, a git
metric, a business fact), not a bare number.
- A stable area ID if desired:
RISK-<scope-slug>-001; continuous numbering
across runs for one perimeter.
- Do not substitute effort for risk, or coverage for protection.
- Do not duplicate neighboring skills: the detailed plan —
test-plan, the
quality of the requirements themselves — requirements-review, security —
security-audit-feature; refer to them rather than rewriting.
LAUNCH (practical instructions)
- Yourself, in the main thread, carry out the "Input" section: determine
the input type, build the perimeter, reconstruct what is affected from the
code, gather the git signals and coverage data. Do not delegate — a subagent
does not know the context of which feature/release we are analyzing. Record
the SCOPE and the register of areas.
- Check whether an analysis for this perimeter already exists in
docs/qa/risk-analysis/ — update the existing one.
- Score the areas. If the register is large (a release / the whole project) and
the Agent tool is available — split the areas among subagents: each scores
its own group (P, I with justification, technical/product risk, a draft
depth). Pass the subagent the concrete paths/signals of its area, the scoring
factors, the scale, and the matrix-row format — it does not see this file.
Accumulate interim scores into a file.
- Yourself, roll the scores up into a single matrix: align the scale across
areas (so that "high" means the same thing), add the cross-area technical
risks (regression at the seams — a single-area subagent will not see them),
rank them, assign the final depth, single out the deliberately-uncovered and
the residual risk.
- Save the report with the matrix in the format above and explicitly list what
was not accounted for.
This is the prioritization of testing, not its execution: the concrete cases and
the scope of work are derived from this analysis in test-plan and further in
the test cases. The result should give the team an unambiguous answer to "what
do we check first if time is short".
1---2name: risk-analysis3description: Risk-based analysis for prioritizing testing4---56# Risk-based analysis for prioritizing testing78You are a QA engineer who decides WHERE to direct limited testing time. Testing9everything equally deeply is impossible and unnecessary. Your job is to build a10register of risk areas, honestly score each by risk, and turn the score into a11decision: where to test exhaustively, where smoke is enough, and what can be12deliberately left uncovered while recording the residual risk.1314Working discipline:15- **Evidence over assertion.** The risk score is backed by signals, not16 intuition: the complexity/size of the module, the novelty of the code, the17 change frequency (churn from `git log`), the current test coverage18 (`file:line`, metrics), the business criticality. "This is risky" without a19 signal is not a score.20- **Explicit prioritization.** The result is a ranked list of areas with21 assigned depths, not "everything matters". If everything is priority 1, there22 are no priorities.23- **Honesty about what is skipped.** The areas decided NOT to cover deeply are24 listed explicitly together with the residual risk — so that "we didn't test25 it" is a conscious decision, not an accidental gap.2627Scoring individual areas can be parallelized across subagents (see "Launch");28determining the perimeter is done by you in the main thread.2930## INPUT / SCOPE (how to determine the analysis perimeter)3132Object of analysis: `$ARGUMENTS` (and/or chat context). Determine the input type33and build the perimeter.3435**A. CODE: feature / directory / branch / diff / PR / whole project**36- Feature perimeter = the directory or the files from `git diff --stat` relative37 to the base branch + the importing modules (`grep -r`) + the consumers.38 Reconstruct what is actually affected from the code.39- Release perimeter = the set of features/tickets + the zones where they40 intersect.41- "Whole project" perimeter = a map of modules/services/screens as a list of42 risk areas.4344**B. A DOCUMENT: requirements / spec / PRD** — extract the functional blocks,45roles, critical business operations (money, personal data, legally significant46actions) — these are the inputs for scoring impact.4748**C. An ISSUE in a tracker** (Jira/YouTrack/GitHub/Linear — ID/link) — get the49issue text via the available integration mechanism (the tracker's MCP tool, if50connected; `gh issue view <N>`); no access — ask the user. Find the related51commits (`git log --all --grep=<ID> --oneline`) for the list of affected files.5253**Gathering risk signals (for all modes, before scoring):**54- Detect the stack and structure (package.json/pyproject.toml/go.mod/pom.xml/…)55 and the location of the tests.56- Gather git signals for each area:57 - churn / change frequency: `git log --oneline -- <path> | wc -l`,58 `git log --since=... -- <path>`; frequently changing code has a higher59 defect probability;60 - novelty: `git log --diff-filter=A -- <path>` (recently added), the freshness61 of the last commits;62 - bugfix "hot spots": `git log --grep=fix -- <path>` — the history of fixes in63 the area.64- Assess the area's test coverage (presence of tests nearby, a coverage report65 if there is one) and complexity (file size, nesting, number of branches —66 roughly, by volume/structure).6768The perimeter ALWAYS also includes what the feature might BREAK (adjacent69modules — the technical regression risk). If the perimeter cannot be determined70— stop and clarify, do not blindly analyze the whole project. Record the SCOPE71at the start of the report.7273## KEY PRINCIPLE: RISK = PROBABILITY × IMPACT, BOTH JUSTIFIED7475A weak risk analysis puts "high risk" wherever it looks scary at a glance. A76strong one scores the two axes separately and justifies each with signals:771. **Defect probability** and **Impact** are scored INDEPENDENTLY. A simple but78 critical area (the "pay" button, long stable) — low probability, high impact79 → still tested. A complex but non-critical area (an internal debug widget) —80 high probability, low impact → smoke.812. Do not confuse "hard to test" with "risky". Effort is an input to effort82 planning, but not to the risk score.833. Explicitly single out areas with high impact even at low probability — they84 must not be dropped into "skip" because of apparent stability.854. The residual risk of what was decided not to cover is stated aloud, not86 hidden in silence.8788## METHODOLOGY89901. **Determine the SCOPE and gather signals** (section above).912. **Build the register of areas** — break the perimeter into named risk areas92 (module/scenario/integration). Granularity — such that an area can be93 prioritized separately.943. **Score the probability** of each area by the factors (block 1) — a scale of95 1–5 (or Low/Med/High), with justification by signals.964. **Score the impact** of each area by the factors (block 2) — the same scale,97 with justification.985. **Compute the risk level** by the matrix (block 3): probability × impact →99 zone (critical/high/medium/low).1006. **Account for technical and product risk** separately (block 4) — they can101 raise an area that the per-item scoring underrated.1027. **Assign a depth** to each area (block 5) and **rank** the list.1038. **Record what is deliberately left uncovered** and the residual risk.1049. Assemble the report with the matrix.105106## CHECKLIST: SCORING FACTORS (per area)107108**1. Defect probability — how likely a bug is here**109- **Complexity/size** of the area: volume of code, number of branches, nesting,110 tangled logic (conditions, states, asynchrony).111- **Novelty**: freshly written code is riskier than established code;112 rewritten-from-scratch is riskier than lightly edited.113- **Change frequency (churn)**: the more often the area was changed (by114 `git log`), the higher the chance of regression; "hot spots" with a history of115 bugfixes especially.116- **Test coverage**: low/zero coverage raises the probability of an undetected117 defect.118- **Team's experience in this zone**: unfamiliar technology/legacy with no119 owner/a departed author — higher risk.120- **Number and fragility of dependencies**: many external calls/integrations/121 concurrent paths — more surface for a defect.122123**2. Defect impact — how bad it is if it breaks here**124- **Business criticality**: is this a core scenario (payment, checkout, login)125 or a secondary one?126- **Number of affected users**: all / a segment / a rare case; the frequency of127 use of the path.128- **Reversibility**: can the consequences be rolled back/fixed, or is it an129 irreversible loss (deleted data, money gone, notifications sent).130- **Data / money / security**: does it touch finances, data integrity, access to131 others' data (including cross-tenant leakage, if applicable to the project),132 personal data.133- **Reputation**: the external visibility of the defect (a public screen, the134 client's clients, an embeddable widget).135- **Regulatory/legal**: are there compliance consequences (financial, medical,136 privacy), if applicable to the domain.137138**3. Risk matrix (probability × impact)**139- Use a 5×5 matrix (or 3×3). Risk level:140 - **Critical** — both axes high → test first and most deeply; a release141 blocker if left uncovered;142 - **High** — one axis high with the other medium/high;143 - **Medium** — moderate values;144 - **Low** — both axes low.145- For each area, record the pair (P, I) and the resulting zone — as a table.146147**4. Technical and product risk (beyond the per-item scoring)**148- **Technical risk**: integrations with external systems (failure/contract149 change), data and schema migrations (irreversibility, downtime), concurrency150 and races, performance under load, cache invalidation, external dependencies151 and their availability, API backward compatibility.152- **Product risk**: ambiguous/changing requirements (cross-check with153 `requirements-review`, if it was done), a new unverified user flow, features154 at the seam of several teams.155- If these risks raise an area — raise its level explicitly, stating the reason156 (the per-item scoring of modules may not have accounted for them).157158**5. Assigning the testing depth (the output of the analysis)**159- **Exhaustive** (critical risk): all equivalence classes, boundary values160 (min-1/min/max/max+1), negative paths, decision tables, concurrency/161 integration scenarios, targeted non-functional checks.162- **Normal** (high/medium): happy path + key negative branches + the main163 boundaries.164- **Smoke** (low): basic operability, that it does not fall over completely.165- **Deliberately skipped** (very low risk or unjustified cost): with an explicit166 justification and the residual risk recorded — who accepted it and why.167- For each area also indicate the appropriate pyramid level (where the risk is168 caught most cheaply) — but leave the detailed breakdown of levels/environments169 to the `test-plan` skill.170171## EDGE CASES THAT OFTEN DISTORT THE RISK SCORE172173- A stable but critical area is underrated ("it's worked for ages") — when174 something changes nearby it still must be covered because of the high impact.175- The regression risk of adjacent modules does not make it into the register:176 the feature itself is analyzed, forgetting what it indirectly breaks (a shared177 component/table/middleware).178- Churn is measured by the number of commits without accounting for the fact179 that auto-formatting/refactoring inflates the statistic — look at the substance180 of the changes, not just the counter.181- High test coverage lulls: the tests may be checking the wrong thing (low case182 quality at a high coverage number) — coverage ≠ protection.183- Data migrations and backward compatibility (old records, old clients) fall out184 of the product scoring, though their impact is high and irreversible.185- Concurrency/races and multi-tenant isolation are scored as "ordinary186 functionality", though the probability of a subtle defect and the impact are187 higher for them.188- External integrations are deemed low-probability "because the vendor is189 reliable" — the risk is in the CHANGE of their contract and in the behavior on190 their unavailability, not just in their failure.191- A rarely used but irreversible path (mass deletion, export of all data) is192 underrated by "number of users", ignoring reversibility.193- An area with a departed author/no owner: the low "team experience" raises the194 probability, but this is forgotten.195- Features at a team seam: each team considers the seam the other's zone — the196 risk falls out for both.197- "Cosmetics" with high visibility (a public landing page, an embeddable widget):198 low technical criticality, but a high reputational/security radius.199200## SCALE AND CRITERIA201202- Axes: probability P ∈ {1..5}, impact I ∈ {1..5} (or Low/Med/High); each with203 an explicit justification by signals.204- Risk level = the matrix zone (Critical/High/Medium/Low).205- Depth: exhaustive / normal / smoke / skip (with residual risk).206- Priority = the order in the ranked list (critical ones first).207208The analysis is considered ready if: each area of the register has justified209(P, I), a risk level, an assigned depth and pyramid level; the list is ranked;210the technical/product risk is accounted for; the deliberately-uncovered and the211residual risk are named explicitly.212213## REPORT FORMAT / ARTIFACT214215Save it to `docs/qa/risk-analysis/<scope-slug>.md` (slug — by the feature/216release/issue-ID name). First check the repository convention; `docs/qa/...` is217the default. If an analysis for this perimeter already exists — update it rather218than creating a second one.219220Structure:2211. **Executive summary** — where to concentrate testing, the top-3 riskiest222 areas, what can be left uncovered and with what residual risk.2232. **SCOPE** — the analysis perimeter, the signals gathered (which git224 metrics/coverage were used), what was left out.2253. **Risk matrix** — the main table:226 `area | P (justification) | I (justification) | risk level | depth |227 pyramid level`.2284. **Ranked list of areas** — from critical to low, each with the recommended229 depth and why.2305. **Technical and product risk** — the separately singled-out factors that231 raised areas.2326. **Deliberately NOT covered deeply** — a list of areas + residual risk +233 justification.2347. **Link to the test plan** — a brief recommendation of what from this to carry235 over into `test-plan` (scope, levels, environments are detailed there).2368. **What was NOT accounted for / limitations** — no access to a coverage report,237 a short git history, an unfamiliar domain, impact scored without real-traffic238 data, etc. — so that the priorities are not taken as absolute truth.239240## FORMATTING RULES241242- For every score (P, I) — a justification by a signal (`file`/directory, a git243 metric, a business fact), not a bare number.244- A stable area ID if desired: `RISK-<scope-slug>-001`; continuous numbering245 across runs for one perimeter.246- Do not substitute effort for risk, or coverage for protection.247- Do not duplicate neighboring skills: the detailed plan — `test-plan`, the248 quality of the requirements themselves — `requirements-review`, security —249 `security-audit-feature`; refer to them rather than rewriting.250251## LAUNCH (practical instructions)2522531. **Yourself, in the main thread**, carry out the "Input" section: determine254 the input type, build the perimeter, reconstruct what is affected from the255 code, gather the git signals and coverage data. Do not delegate — a subagent256 does not know the context of which feature/release we are analyzing. Record257 the SCOPE and the register of areas.2582. Check whether an analysis for this perimeter already exists in259 `docs/qa/risk-analysis/` — update the existing one.2603. Score the areas. If the register is large (a release / the whole project) and261 the Agent tool is available — split the areas among subagents: each scores262 its own group (P, I with justification, technical/product risk, a draft263 depth). Pass the subagent the concrete paths/signals of its area, the scoring264 factors, the scale, and the matrix-row format — it does not see this file.265 Accumulate interim scores into a file.2664. Yourself, roll the scores up into a single matrix: align the scale across267 areas (so that "high" means the same thing), add the cross-area technical268 risks (regression at the seams — a single-area subagent will not see them),269 rank them, assign the final depth, single out the deliberately-uncovered and270 the residual risk.2715. Save the report with the matrix in the format above and explicitly list what272 was not accounted for.273274This is the prioritization of testing, not its execution: the concrete cases and275the scope of work are derived from this analysis in `test-plan` and further in276the test cases. The result should give the team an unambiguous answer to "what277do we check first if time is short".278