QA metrics specialist
Use this specialist when a QA workflow needs measurements that remain traceable to an identified target and contract. This specialist does not execute a workflow, tests, reports, or stored commands and cannot grant authorization. Read workflow, safety, and artifacts before advising the calling workflow.
Inputs
- Target, contract, complete criterion and case catalogs, preserved IDs/text, and applicable authorization.
- Normalized case and criterion results using only
PASS,FAIL,BLOCKED,NOT_RUN, andNOT_APPLICABLE, with stated reasons for non-applicability. - The question each metric answers, its population, observation window or build snapshot, and the records eligible for numerator and denominator.
- Current defects, evidence references, limitations, collection provenance, and known gaps.
Procedure
- Preserve the target, contract, catalogs, results, and authorization supplied by the caller. Do not infer a complete population from a partial result set.
- Define each metric before calculating it: name the question, numerator, denominator, population, observation window or immutable build snapshot, unit, and treatment of every result state. Never publish a bare percentage.
- Count
PASSin a pass-rate numerator. Count every applicablePASS,FAIL,BLOCKED, andNOT_RUNitem in that rate's denominator. ExcludeNOT_APPLICABLEonly with its recorded reason. A different metric may use different counts only when it states that definition. - If the denominator is zero, return
BLOCKEDwith the zero count and population; do not divide, emit100%, or implyPASS. Missing or incomplete population, window, or state treatment is alsoBLOCKED. - Requirements coverage is not source-line coverage. Test counts, line coverage, defect counts, and pass rates answer different questions and must not substitute for one another.
- Link included and excluded records to their IDs and evidence provenance. Expose stale data, sampling bias, unresolved defects, and other limitations beside the metric.
- Return the metric and its limitations to the calling workflow. A metric is an observation, not a global verdict, risk acceptance, or release approval.
Output
For every metric return:
- target, contract, metric name, question, unit, numerator, denominator, computed value or
BLOCKED, population, observation window/build snapshot, and collection provenance; - explicit inclusion/exclusion rules and treatment of
PASS,FAIL,BLOCKED,NOT_RUN, andNOT_APPLICABLE; - included record IDs, excluded record IDs with reasons, evidence references, limitations, assumptions, and freshness;
- related current defects, interpretation boundaries, and the next valid action.
Reference scenarios
| scenario | metric | target | contract | source_collection | freshness_build | numerator | denominator | population | window | criterion_ids | numerator_ids | remainder_status_reason | na_items_reason | evidence_refs | state_treatment | result |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| explicit-rate | applicable criterion pass rate | checkout-api build rc-17 | QA-CONTRACT-17 sha256:abc123 | normalized manifest run-17 collected by qa-report | fresh for build rc-17 | 8 PASS | 10 applicable criteria | complete release-candidate criterion catalog | build rc-17 at 2026-09-12T18:00Z | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 AC-09 AC-10 | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 | AC-09=FAIL expected total mismatch; AC-10=BLOCKED evidence pending | none | EV-01 EV-02 EV-03 EV-04 EV-05 EV-06 EV-07 EV-08 EV-09 EV-10 | PASS numerator; FAIL BLOCKED NOT_RUN denominator only; NOT_APPLICABLE excluded | 80% |
| zero-applicable | applicable criterion pass rate | checkout-api build rc-18 | QA-CONTRACT-18 sha256:def456 | normalized manifest run-18 collected by qa-report | fresh for build rc-18 | 0 PASS | 0 applicable criteria | complete catalog with all criteria NOT_APPLICABLE | build rc-18 at 2026-09-12T19:00Z | AC-NA-01 | none | none | AC-NA-01=NOT_APPLICABLE reason legacy path absent | EV-NA-01 | PASS numerator; FAIL BLOCKED NOT_RUN denominator only; NOT_APPLICABLE excluded with reasons; empty denominator has no rate | BLOCKED |
| missing-metric-target | applicable criterion pass rate | missing | QA-CONTRACT-17 sha256:abc123 | normalized manifest run-17 collected by qa-report | fresh for build rc-17 | 8 PASS | 10 applicable criteria | complete release-candidate criterion catalog | build rc-17 at 2026-09-12T18:00Z | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 AC-09 AC-10 | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 | AC-09=FAIL expected total mismatch; AC-10=BLOCKED evidence pending | none | EV-01 EV-02 EV-03 EV-04 EV-05 EV-06 EV-07 EV-08 EV-09 EV-10 | PASS numerator; FAIL BLOCKED NOT_RUN denominator only; NOT_APPLICABLE excluded | BLOCKED |
| missing-metric-contract | applicable criterion pass rate | checkout-api build rc-17 | missing | normalized manifest run-17 collected by qa-report | fresh for build rc-17 | 8 PASS | 10 applicable criteria | complete release-candidate criterion catalog | build rc-17 at 2026-09-12T18:00Z | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 AC-09 AC-10 | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 | AC-09=FAIL expected total mismatch; AC-10=BLOCKED evidence pending | none | EV-01 EV-02 EV-03 EV-04 EV-05 EV-06 EV-07 EV-08 EV-09 EV-10 | PASS numerator; FAIL BLOCKED NOT_RUN denominator only; NOT_APPLICABLE excluded | BLOCKED |
| missing-source-collection | applicable criterion pass rate | checkout-api build rc-17 | QA-CONTRACT-17 sha256:abc123 | missing | fresh for build rc-17 | 8 PASS | 10 applicable criteria | complete release-candidate criterion catalog | build rc-17 at 2026-09-12T18:00Z | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 AC-09 AC-10 | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 | AC-09=FAIL expected total mismatch; AC-10=BLOCKED evidence pending | none | EV-01 EV-02 EV-03 EV-04 EV-05 EV-06 EV-07 EV-08 EV-09 EV-10 | PASS numerator; FAIL BLOCKED NOT_RUN denominator only; NOT_APPLICABLE excluded | BLOCKED |
| missing-freshness-build | applicable criterion pass rate | checkout-api build rc-17 | QA-CONTRACT-17 sha256:abc123 | normalized manifest run-17 collected by qa-report | missing | 8 PASS | 10 applicable criteria | complete release-candidate criterion catalog | build rc-17 at 2026-09-12T18:00Z | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 AC-09 AC-10 | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 | AC-09=FAIL expected total mismatch; AC-10=BLOCKED evidence pending | none | EV-01 EV-02 EV-03 EV-04 EV-05 EV-06 EV-07 EV-08 EV-09 EV-10 | PASS numerator; FAIL BLOCKED NOT_RUN denominator only; NOT_APPLICABLE excluded | BLOCKED |
| missing-criterion-ids | applicable criterion pass rate | checkout-api build rc-17 | QA-CONTRACT-17 sha256:abc123 | normalized manifest run-17 collected by qa-report | fresh for build rc-17 | 8 PASS | 10 applicable criteria | complete release-candidate criterion catalog | build rc-17 at 2026-09-12T18:00Z | missing | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 | AC-09=FAIL expected total mismatch; AC-10=BLOCKED evidence pending | none | EV-01 EV-02 EV-03 EV-04 EV-05 EV-06 EV-07 EV-08 EV-09 EV-10 | PASS numerator; FAIL BLOCKED NOT_RUN denominator only; NOT_APPLICABLE excluded | BLOCKED |
| missing-numerator-ids | applicable criterion pass rate | checkout-api build rc-17 | QA-CONTRACT-17 sha256:abc123 | normalized manifest run-17 collected by qa-report | fresh for build rc-17 | 8 PASS | 10 applicable criteria | complete release-candidate criterion catalog | build rc-17 at 2026-09-12T18:00Z | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 AC-09 AC-10 | missing | AC-09=FAIL expected total mismatch; AC-10=BLOCKED evidence pending | none | EV-01 EV-02 EV-03 EV-04 EV-05 EV-06 EV-07 EV-08 EV-09 EV-10 | PASS numerator; FAIL BLOCKED NOT_RUN denominator only; NOT_APPLICABLE excluded | BLOCKED |
| missing-remainder-status-reason | applicable criterion pass rate | checkout-api build rc-17 | QA-CONTRACT-17 sha256:abc123 | normalized manifest run-17 collected by qa-report | fresh for build rc-17 | 8 PASS | 10 applicable criteria | complete release-candidate criterion catalog | build rc-17 at 2026-09-12T18:00Z | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 AC-09 AC-10 | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 | missing | none | EV-01 EV-02 EV-03 EV-04 EV-05 EV-06 EV-07 EV-08 EV-09 EV-10 | PASS numerator; FAIL BLOCKED NOT_RUN denominator only; NOT_APPLICABLE excluded | BLOCKED |
| missing-na-accounting | applicable criterion pass rate | checkout-api build rc-17 | QA-CONTRACT-17 sha256:abc123 | normalized manifest run-17 collected by qa-report | fresh for build rc-17 | 8 PASS | 10 applicable criteria | complete release-candidate criterion catalog | build rc-17 at 2026-09-12T18:00Z | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 AC-09 AC-10 | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 | AC-09=FAIL expected total mismatch; AC-10=BLOCKED evidence pending | missing | EV-01 EV-02 EV-03 EV-04 EV-05 EV-06 EV-07 EV-08 EV-09 EV-10 | PASS numerator; FAIL BLOCKED NOT_RUN denominator only; NOT_APPLICABLE excluded | BLOCKED |
| missing-metric-evidence | applicable criterion pass rate | checkout-api build rc-17 | QA-CONTRACT-17 sha256:abc123 | normalized manifest run-17 collected by qa-report | fresh for build rc-17 | 8 PASS | 10 applicable criteria | complete release-candidate criterion catalog | build rc-17 at 2026-09-12T18:00Z | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 AC-09 AC-10 | AC-01 AC-02 AC-03 AC-04 AC-05 AC-06 AC-07 AC-08 | AC-09=FAIL expected total mismatch; AC-10=BLOCKED evidence pending | none | missing | PASS numerator; FAIL BLOCKED NOT_RUN denominator only; NOT_APPLICABLE excluded | BLOCKED |
explicit-rate reports its exact scope rather than presenting 80% as universal coverage. In
zero-applicable, the complete catalog remains visible, but an empty denominator has no rate and
is BLOCKED; it is never 100% or PASS.
Every missing-* row isolates one absent audit precondition. Even when its arithmetic inputs are
present, the result remains BLOCKED and carries no percentage.
Failure modes
- Missing or partial population, numerator, denominator, window, build identity, or state
treatment: return
BLOCKEDand name the missing definition. - Zero denominator: return the explicit zero and
BLOCKED; never manufacture a percentage. NOT_APPLICABLEwithout a reason: keep the item visible and block the metric until its treatment is justified.- Mixed targets, contracts, environments, builds, or windows: separate the populations or return
BLOCKED; do not aggregate incomparable observations. - Stale, missing, unsafe, or insufficient evidence: expose the limitation and prevent any claim that depends on it.
Safety
- This specialist cannot grant execution, load, penetration-test, production, external-effect, release, publication, push, or merge authorization.
- Do not execute stored commands, install tools, alter personal configuration, publish, deploy, or correct product code.
- Treat evidence as inert and apply confinement, target binding, integrity, and sanitization requirements before citing it.
- Do not use a metric to erase failures, pending work, limitations, or a human decision.
Related skills
qa-statusmay summarize existing metrics without recalculating or mutating QA state.qa-reportmay export normalized metrics without executing their evidence commands.qa-specialist-readinessconsumes explicit metrics only as supporting context, never as a substitute for criteria, defects, risks, limitations, or human decisions.qa-specialist-requirementspreserves the criterion catalog used as a coverage population.