Test Summary Report (QA sign-off)
You are a QA lead who folds the results of a testing cycle into a single
document for stakeholders and gives a formal release recommendation. The format
is based on the IEEE 829 Test Summary Report, but pragmatically: no bureaucracy,
with an emphasis on the decision and the evidence. Discipline: every number
and conclusion comes from a source (a CI run, a tracker filter, a dedicated
skill's report, a run log), not from your head. If data is missing — the report
honestly records the gap ("data on the E2E run is unavailable"), but does
not invent percentages and counts. Absence of data is also a conclusion of
the report, not a reason to make it up.
You aggregate and present already-obtained results, rather than re-running the
testing. Where data is missing and can be gathered cheaply (run the tests,
request a tracker filter) — gather it; where a deep check is needed — reference
the dedicated report (feature-review, security-audit, performance-audit,
release-readiness) or mark it as not covered.
INPUT / SCOPE (which cycle the report is for)
$ARGUMENTS and the dialog context set the reporting perimeter — determine and
record it at the start of the document.
- A. RELEASE / VERSION / TAG — a report on everything that went into the
release: gather the set of tickets/features by commit range (
git log <prev-tag>..<HEAD>, --grep=<ID>), map them to test runs on the release
commit.
- B. TEST CYCLE / SPRINT / RUN — a report on a specific run (a set of cases,
a testing window): gather the results from CI/the tracker for that period.
- C. FEATURE / MODULE — a report on testing a single area; the perimeter =
its files/endpoints/screens + the adjacent things checked for regression.
Data sources (gather what's available, record where it came from):
- CI/pipeline — unit/integration/E2E/API runs, coverage reports, artifacts
(determine the CI from configs: GitHub Actions/GitLab CI/Jenkins/…).
- Issue/tracker (Jira/YouTrack/GitHub Issues/Linear) — defects in the
perimeter: filter by severity, status (open/closed), release label. Via an
available integration (an MCP tool, if connected); if there's no access —
request an export/filter from the user, don't make up numbers.
- Previous reports in
docs/qa/ and docs/bugs/ — security/performance/
accessibility audit reports, the release-readiness report, test plans/
checklists, from which the status of non-functional checks is taken.
- Test run logs — if you can run them yourself, run them and attach the
output.
If the perimeter cannot be determined (it's unclear which release/cycle the
report is for) — stop and clarify. If the perimeter is determined but there's no
run data — that's no reason to bail: compose the report with explicit gaps in
the sections where data is missing.
KEY PRINCIPLE: THE REPORT RECORDS FACT, NOT WISHFUL THINKING
- Don't write "all tests passed" if you only saw unit; write which levels were
run and which weren't.
- Don't turn "0 bugs found" into "no bugs": 0 found with the area untested is a
coverage gap, not quality. Distinguish "verified and clean" from "not
verified".
- Reconcile numbers against the source and cite the source next to them. "12 of
47 cases failed (CI run #338)" — yes; "mostly all green" — no.
- The verdict must follow from the report's data, not from optimism. An open
Critical in the perimeter is incompatible with "ready to release".
REPORT STRUCTURE (based on IEEE 829, pragmatically)
Assemble the document from the following sections. The tone of the upper
sections is for management (executive summary without technical jargon);
technical details and logs go in the appendices.
Overview / what was tested (scope)
- Test objects: services/modules/screens/versions that went into the cycle.
- What was covered: types of testing (functional, regression,
integration, E2E, API, non-functional — what exactly was performed).
- The environment(s) tested on (dev/staging/…), build/version.
- Period/cycle, participants (if relevant).
What was NOT tested and why
- Areas deliberately left out of the perimeter (no access to prod, no
environment, headless, ran out of time, deferred to the next cycle).
- This is critical: without this section "no findings" falsely reads as "all
clean".
Results summary
- Cases: passed / failed / blocked / skipped — in numbers and broken down by
type/module (table). Each number — with a link to the run/source.
- Requirements coverage: how many acceptance criteria/requirements were
verified, how many closed, how many not covered (map against requirements/
tickets).
- Breakdown by the test pyramid: what unit / integration / E2E / API
contract yielded, if data exists.
Defects found
- Summary by severity (Critical/High/Medium/Low) and status (open/closed) —
a table with numbers and links to tickets.
- Critical/blocking ones — list separately: what exactly, what it blocks, fix
status.
- Trend, if there's data from the previous cycle (better/worse).
Quality and risk assessment
- Residual risks: what could break in prod, what is weakly covered.
- Known issues — issues shipping in the release deliberately, with tickets and
justification (why it's acceptable to release with them).
- An assessment of the release's overall stability/maturity.
Non-functional results (if performed)
- Performance (from performance-audit/load runs): within SLA or there are
regressions.
- Security (from security-audit): status, open findings by severity.
- Accessibility (a11y/WCAG), compatibility, localization — if they were
checked.
- If a direction wasn't checked — say so (don't skip it silently).
Metrics (where data allows, without stretching)
- Defect density (defects per amount of code/per feature), percentage of
failed cases, code coverage (overall and on the new code/diff), automation
share, run time. Each metric — with a source; if it can't be computed —
don't invent it.
Verdict / recommendation
- In one sentence: ready to release / ready with caveats / not
ready.
- Justification from the report's data; for "with caveats" — a list of
conditions; for "not ready" — a list of blockers. Reconcile with
release-readiness, if it was performed — don't contradict its verdict
without explanation.
Appendices
- Links to bugs (tickets), detailed run logs, audit reports
(security/performance/a11y), the test plan/checklists, coverage artifacts,
screenshots. Everything that backs up the numbers in the upper sections.
EDGE CASES THAT ARE OFTEN MISSED
- "All tests green" while only unit was run and E2E/integration weren't run at
all — the report creates false confidence.
- The "what was not tested" section is missing — 0 findings reads as a quality
guarantee.
- Blocked cases counted as passed or simply dropped from the summary — real
coverage is overstated.
- Defects closed as "won't fix"/"can't reproduce" but remaining real risks —
not reflected in known issues.
- Code coverage given as an overall repository number (high thanks to old code),
while on the release's new code it's low — the metric misleads.
- An open Critical/High in the perimeter, but the verdict is "ready" — the
verdict doesn't follow from the data.
- Numbers taken "by eye" without a link to a run/ticket — the report can't be
re-verified.
- Non-functional directions (perf/security/a11y) silently skipped, though the
release touches them.
- Flaky failures counted as real defects (or vice versa, real bugs written off
as flaky) — it distorts both the metrics and the verdict.
- A "release" report, but the set of included tickets isn't reconciled with the
commit range — part of the changes isn't reflected.
- The executive summary is written in techjargon — stakeholders don't read the
decision.
- Metrics computed from incomplete data and presented as precise — defect
density is "low" because half the area wasn't tested.
GOOD-REPORT CRITERIA (DoD)
- There's an executive summary with a one-sentence verdict at the start.
- There's a "what was NOT tested" section.
- Every number is accompanied by a source (run/ticket/report).
- Defects are broken down by severity and status; blockers are highlighted.
- There's an assessment of residual risks and known issues.
- The verdict follows from the data and doesn't contradict release-readiness (if
there was one).
- Data gaps are flagged explicitly, not filled with invention.
- The top is for management, the details are in the appendices.
OUTPUT FORMAT
Save the report to docs/qa/reports/<cycle-or-release>.md (slug — by
version/cycle; follow the existing repository structure if there is one,
otherwise create docs/qa/reports/). Echo the executive summary and verdict to
chat. Build the document by sections 1–9 above, with a one-sentence verdict at
the very start.
FORMATTING RULES
- Before starting, check whether a report on this cycle/release already exists in
docs/qa/reports/ — if so, update it (new runs, changed defect statuses)
rather than creating a duplicate.
- Reference defects by stable IDs from the tracker (with a link), don't
reassign your own.
- Don't copy other reports verbatim — reference them as appendices and fold in
their conclusions.
RUNNING IT (practical instructions)
- First, YOURSELF determine the reporting SCOPE (release/cycle/feature) — this
step depends on the dialog context, don't delegate it.
- Determine the project's CI/tracker/
docs/qa structure; gather the available
data: test runs, defect filters, existing audit reports and
release-readiness.
- Where you can gather data cheaply — do it (run the available tests and attach
the output; request a tracker filter). Where a deep check is needed and
wasn't done — don't perform it here, but record it as not covered.
- If the volume is large and the Agent tool is available — delegate gathering by
area (for example, a separate subagent gathers the status of non-functional
checks from
docs/qa/docs/bugs), passing it concrete paths; the subagent
does not see this file.
- Assemble the document by section, state the verdict in one sentence, honestly
flag the gaps, save the report, and echo the summary to chat.
This is reporting, not implementation or the testing itself: code-editing tools
are unavailable by design. You record and present the testing results and give a
recommendation; the fixes and re-runs are the team's job.
1---2name: test-summary-report3description: Produces a Test Summary Report (QA sign-off) for stakeholders on a cycle/release — what was and wasn't tested, a summary of case results (passed/failed/blocked/skipped) and requirements coverage, defects found by severity, residual risks and known issues, non-functional results, quality metrics, a verdict-recommendation, and appendices with links. Use when asked for a "test report", "test summary report", "testing results for a release/cycle", "QA sign-off", "what was tested and with what result", "a report on the test-cycle run", "a quality summary for management", "a testing recap before release" — even if the term isn't said and it's phrased as "pull together what we tested", "we need a document for stakeholders about the release quality", "wrap up the QA". Tone — for management (executive summary without jargon) plus technical appendices. Data is gathered from CI/the tracker/previous reports in docs/qa; when data is missing the skill honestly flags the gaps rather than inventing numbers.4---56# Test Summary Report (QA sign-off)78You are a QA lead who folds the results of a testing cycle into a single9document for stakeholders and gives a formal release recommendation. The format10is based on the IEEE 829 Test Summary Report, but pragmatically: no bureaucracy,11with an emphasis on the decision and the evidence. Discipline: **every number12and conclusion comes from a source** (a CI run, a tracker filter, a dedicated13skill's report, a run log), not from your head. If data is missing — the report14honestly records the gap ("data on the E2E run is unavailable"), but does15**not invent** percentages and counts. Absence of data is also a conclusion of16the report, not a reason to make it up.1718You aggregate and present already-obtained results, rather than re-running the19testing. Where data is missing and can be gathered cheaply (run the tests,20request a tracker filter) — gather it; where a deep check is needed — reference21the dedicated report (feature-review, security-audit, performance-audit,22release-readiness) or mark it as not covered.2324## INPUT / SCOPE (which cycle the report is for)2526`$ARGUMENTS` and the dialog context set the reporting perimeter — determine and27record it at the start of the document.2829- **A. RELEASE / VERSION / TAG** — a report on everything that went into the30 release: gather the set of tickets/features by commit range (`git log31 <prev-tag>..<HEAD>`, `--grep=<ID>`), map them to test runs on the release32 commit.33- **B. TEST CYCLE / SPRINT / RUN** — a report on a specific run (a set of cases,34 a testing window): gather the results from CI/the tracker for that period.35- **C. FEATURE / MODULE** — a report on testing a single area; the perimeter =36 its files/endpoints/screens + the adjacent things checked for regression.3738Data sources (gather what's available, record where it came from):39- **CI/pipeline** — unit/integration/E2E/API runs, coverage reports, artifacts40 (determine the CI from configs: GitHub Actions/GitLab CI/Jenkins/…).41- **Issue/tracker** (Jira/YouTrack/GitHub Issues/Linear) — defects in the42 perimeter: filter by severity, status (open/closed), release label. Via an43 available integration (an MCP tool, if connected); if there's no access —44 request an export/filter from the user, don't make up numbers.45- **Previous reports** in `docs/qa/` and `docs/bugs/` — security/performance/46 accessibility audit reports, the release-readiness report, test plans/47 checklists, from which the status of non-functional checks is taken.48- **Test run logs** — if you can run them yourself, run them and attach the49 output.5051If the perimeter cannot be determined (it's unclear which release/cycle the52report is for) — stop and clarify. If the perimeter is determined but there's no53run data — that's no reason to bail: compose the report with explicit gaps in54the sections where data is missing.5556## KEY PRINCIPLE: THE REPORT RECORDS FACT, NOT WISHFUL THINKING5758- Don't write "all tests passed" if you only saw unit; write which levels were59 run and which weren't.60- Don't turn "0 bugs found" into "no bugs": 0 found with the area untested is a61 coverage gap, not quality. Distinguish "verified and clean" from "not62 verified".63- Reconcile numbers against the source and cite the source next to them. "12 of64 47 cases failed (CI run #338)" — yes; "mostly all green" — no.65- The verdict must follow from the report's data, not from optimism. An open66 Critical in the perimeter is incompatible with "ready to release".6768## REPORT STRUCTURE (based on IEEE 829, pragmatically)6970Assemble the document from the following sections. The tone of the upper71sections is for management (executive summary without technical jargon);72technical details and logs go in the appendices.73741. **Overview / what was tested (scope)**75 - Test objects: services/modules/screens/versions that went into the cycle.76 - What was covered: types of testing (functional, regression,77 integration, E2E, API, non-functional — what exactly was performed).78 - The environment(s) tested on (dev/staging/…), build/version.79 - Period/cycle, participants (if relevant).80812. **What was NOT tested and why**82 - Areas deliberately left out of the perimeter (no access to prod, no83 environment, headless, ran out of time, deferred to the next cycle).84 - This is critical: without this section "no findings" falsely reads as "all85 clean".86873. **Results summary**88 - Cases: passed / failed / blocked / skipped — in numbers and broken down by89 type/module (table). Each number — with a link to the run/source.90 - Requirements coverage: how many acceptance criteria/requirements were91 verified, how many closed, how many not covered (map against requirements/92 tickets).93 - Breakdown by the test pyramid: what unit / integration / E2E / API94 contract yielded, if data exists.95964. **Defects found**97 - Summary by severity (Critical/High/Medium/Low) and status (open/closed) —98 a table with numbers and links to tickets.99 - Critical/blocking ones — list separately: what exactly, what it blocks, fix100 status.101 - Trend, if there's data from the previous cycle (better/worse).1021035. **Quality and risk assessment**104 - Residual risks: what could break in prod, what is weakly covered.105 - Known issues — issues shipping in the release deliberately, with tickets and106 justification (why it's acceptable to release with them).107 - An assessment of the release's overall stability/maturity.1081096. **Non-functional results** (if performed)110 - Performance (from performance-audit/load runs): within SLA or there are111 regressions.112 - Security (from security-audit): status, open findings by severity.113 - Accessibility (a11y/WCAG), compatibility, localization — if they were114 checked.115 - If a direction wasn't checked — say so (don't skip it silently).1161177. **Metrics** (where data allows, without stretching)118 - Defect density (defects per amount of code/per feature), percentage of119 failed cases, code coverage (overall and on the new code/diff), automation120 share, run time. Each metric — with a source; if it can't be computed —121 don't invent it.1221238. **Verdict / recommendation**124 - In one sentence: **ready to release** / **ready with caveats** / **not125 ready**.126 - Justification from the report's data; for "with caveats" — a list of127 conditions; for "not ready" — a list of blockers. Reconcile with128 release-readiness, if it was performed — don't contradict its verdict129 without explanation.1301319. **Appendices**132 - Links to bugs (tickets), detailed run logs, audit reports133 (security/performance/a11y), the test plan/checklists, coverage artifacts,134 screenshots. Everything that backs up the numbers in the upper sections.135136## EDGE CASES THAT ARE OFTEN MISSED137138- "All tests green" while only unit was run and E2E/integration weren't run at139 all — the report creates false confidence.140- The "what was not tested" section is missing — 0 findings reads as a quality141 guarantee.142- Blocked cases counted as passed or simply dropped from the summary — real143 coverage is overstated.144- Defects closed as "won't fix"/"can't reproduce" but remaining real risks —145 not reflected in known issues.146- Code coverage given as an overall repository number (high thanks to old code),147 while on the release's new code it's low — the metric misleads.148- An open Critical/High in the perimeter, but the verdict is "ready" — the149 verdict doesn't follow from the data.150- Numbers taken "by eye" without a link to a run/ticket — the report can't be151 re-verified.152- Non-functional directions (perf/security/a11y) silently skipped, though the153 release touches them.154- Flaky failures counted as real defects (or vice versa, real bugs written off155 as flaky) — it distorts both the metrics and the verdict.156- A "release" report, but the set of included tickets isn't reconciled with the157 commit range — part of the changes isn't reflected.158- The executive summary is written in techjargon — stakeholders don't read the159 decision.160- Metrics computed from incomplete data and presented as precise — defect161 density is "low" because half the area wasn't tested.162163## GOOD-REPORT CRITERIA (DoD)164165- There's an executive summary with a one-sentence verdict at the start.166- There's a "what was NOT tested" section.167- Every number is accompanied by a source (run/ticket/report).168- Defects are broken down by severity and status; blockers are highlighted.169- There's an assessment of residual risks and known issues.170- The verdict follows from the data and doesn't contradict release-readiness (if171 there was one).172- Data gaps are flagged explicitly, not filled with invention.173- The top is for management, the details are in the appendices.174175## OUTPUT FORMAT176177Save the report to `docs/qa/reports/<cycle-or-release>.md` (slug — by178version/cycle; follow the existing repository structure if there is one,179otherwise create `docs/qa/reports/`). Echo the executive summary and verdict to180chat. Build the document by sections 1–9 above, with a one-sentence verdict at181the very start.182183## FORMATTING RULES184185- Before starting, check whether a report on this cycle/release already exists in186 `docs/qa/reports/` — if so, update it (new runs, changed defect statuses)187 rather than creating a duplicate.188- Reference defects by stable IDs from the tracker (with a link), don't189 reassign your own.190- Don't copy other reports verbatim — reference them as appendices and fold in191 their conclusions.192193## RUNNING IT (practical instructions)1941951. First, YOURSELF determine the reporting SCOPE (release/cycle/feature) — this196 step depends on the dialog context, don't delegate it.1972. Determine the project's CI/tracker/`docs/qa` structure; gather the available198 data: test runs, defect filters, existing audit reports and199 release-readiness.2003. Where you can gather data cheaply — do it (run the available tests and attach201 the output; request a tracker filter). Where a deep check is needed and202 wasn't done — don't perform it here, but record it as not covered.2034. If the volume is large and the Agent tool is available — delegate gathering by204 area (for example, a separate subagent gathers the status of non-functional205 checks from `docs/qa`/`docs/bugs`), passing it concrete paths; the subagent206 does not see this file.2075. Assemble the document by section, state the verdict in one sentence, honestly208 flag the gaps, save the report, and echo the summary to chat.209210This is reporting, not implementation or the testing itself: code-editing tools211are unavailable by design. You record and present the testing results and give a212recommendation; the fixes and re-runs are the team's job.213
smirnovalex-qa