Product Goal-Based Audit
Systematic quality audit that extracts design expectations from project docs,
confirms the project's actual commitments, configures expert audit roles,
generates executable checks, scores the system, cross-validates findings,
reviews the audit itself, and drives iterative improvement through
audit-diagnose-fix-verify cycles.
Stance
Forensic auditor who trusts data over claims. Measures what the system actually does
against what it was designed to do. Treats PASS as "not yet proven wrong" rather than
"confirmed working." Does not accept surface-level scores: distribution, trends,
causality, expert disagreement, and missing evidence matter more than aggregate
PASS/FAIL.
Subcommands
| Command |
Phase(s) |
Description |
mine |
1 |
Extract expectations from project docs |
instrument |
2 |
Generate audit scripts from expectations |
audit |
3 |
Run audit, produce scored report |
deep-dive |
4 |
Analyze distributions, trends, causality beyond PASS/FAIL |
diagnose |
5 |
Identify root causes, build causal chains |
prescribe |
6 |
Generate structured fix prompts with dependency ordering |
verify |
7 |
Re-audit after fixes, compare scores, check for regressions |
full |
1-7 |
Complete audit cycle with iteration |
Default behavior: If no subcommand, check project state:
- No
EXPECTATIONS.md exists → start with mine
- Has expectations but no audit script → start with
instrument
- Has audit script → start with
audit
- Has recent audit report → ask user which phase to enter
Audit directory: Unless the user provides another location, write audit
artifacts under {project-path}/.product-audit/:
.product-audit/COMMITMENT_REVIEW.md
.product-audit/EXPECTATIONS.md
.product-audit/audit.sh or equivalent generated checker
.product-audit/AUDIT_SCRIPT_REVIEW.md
.product-audit/AUDIT_RESULT_REVIEW.md
.product-audit/reports/
.product-audit/state.md for iteration history, hashes, active roles, and latest report paths
Commitment Confirmation And Audit Self-Review
Before finalizing expectations, mine commitment candidates into
COMMITMENT_REVIEW.md, classify each claim, self-review the mining result, and
ask the user to confirm the real current commitments. If the user requests a
trial audit before confirmation, mark the scope as provisional in
expectations, reports, and state.
Before trusting generated audit infrastructure, review the audit itself:
- Phase 1: commitment mining self-review
- Phase 2:
AUDIT_SCRIPT_REVIEW.md
- Phase 3:
AUDIT_RESULT_REVIEW.md
Audit outcomes must be labelled trusted, provisional, or blocked.
Self-review is adversarial confidence building, not proof of perfect coverage.
Adversarial Benchmark Mode
Use adversarial benchmark cases when changing the audit workflow or validating
audit-of-audit behavior. The benchmark suite lives under
benchmarks/adversarial/ and must include cases for roadmap traps, marketing
traps, hidden critical promises, conflicting docs, fake PASS scripts, CRITICAL
downgrades, manual coverage inflation, cumulative metric traps, pretty but wrong
reports, expert disagreement, document-only specs, and external dependency
boundaries.
Expert Panel Mode
When the audit goal or artifact set is broad enough, configure an expert audit
panel before generating expectations. Load references/expert-role-taxonomy.md
to select roles, then load references/multi-agent-orchestration.md when the
runtime can run subagents or when sequential independent passes are useful.
Typical roles include product/requirements, architecture, engineering,
test strategy, security, reliability/operations, performance, data,
UX/accessibility, delivery/DevOps, compliance/privacy, documentation/DX,
and AI behavior.
Expert panel outputs are evidence, not decoration:
- Role coverage summary
- Finding matrix with role, evidence, severity, confidence, and cross-check status
- Duplicate review results for CRITICAL or ambiguous expectations
- Disagreement ledger with adjudication
- Integrated risk list carried into diagnosis and prescriptions
Red Lines
- No vague expectations — Every expectation must have a quantifiable threshold. Reject "should work well."
- No uncovered expectations — Automated checks must cover >= 95% of defined expectations. Manual/INFO stubs are documented gaps and do not count toward automated coverage.
- No incomplete fix prompts — Each must have: current data, source paths, verification method, target value, dependencies.
- No monitor tampering — Audit infrastructure (scripts, expectations, reports) is read-only during fix execution.
- No skipped gates — Each phase transition requires its gate checklist to pass.
- No silent CRITICAL downgrades — A CRITICAL severity cannot become INFO without an explicit Condition field.
- No cumulative-only traps — Expectations for ongoing operations must use
incremental metric type, not cumulative.
- No unbounded iteration — Maximum 5 audit-fix rounds without explicit user override.
- No unconfirmed commitment scope — Final expectations require user-confirmed commitments or visible
provisional status.
- No trusted audit without audit-of-audit — Script and report self-reviews must pass before diagnosis or prescriptions rely on a report.
Scoring Overview
- Weights: CRITICAL=3x, HIGH=2x, MEDIUM=1x, LOW=0.5x
- Results: PASS=full, WARN=half, FAIL=0, INFO=excluded
- CRITICAL blocker: Any CRITICAL FAIL → grade capped at C maximum
- Grades: A>=90, B>=75, C>=60, D>=40, F<40 (A/B require 0 CRITICAL FAILs)
- Full formula:
references/scoring-formula.md
Acceptance Criteria
- Generated EXPECTATIONS.md covers confirmed extractable commitments (coverage manifest produced)
- Commitment confirmation is recorded, or the audit is visibly marked
provisional
- Audit script runs without errors on first invocation and produces a scored report
- Script and result self-reviews pass before diagnosis or prescriptions trust the report
- After 1-3 fix rounds, measurable score improvement with no undetected regressions
- All 12 review findings from the original brief are addressed (see Finding Map below)
Phase Modules (Lazy-Loaded)
When entering a phase, Read the corresponding file before executing any phase logic.
| Phase |
File |
Key Deliverable |
| 1: Mine |
phases/phase-1-mine.md |
EXPECTATIONS.md + coverage manifest + audit panel |
| 2: Instrument |
phases/phase-2-instrument.md |
Audit script + coverage verification + self-test |
| 3: Audit |
phases/phase-3-audit.md |
Scored report + expert finding matrix |
| 4: Deep Dive |
phases/phase-4-deep-dive.md |
"Surface PASS, Deep Concern" findings + cross-validation |
| 5: Diagnose |
phases/phase-5-diagnose.md |
Causal graph + role-attributed root causes |
| 6: Prescribe |
phases/phase-6-prescribe.md |
Structured fix prompts with dependency ordering and role evidence |
| 7: Verify |
phases/phase-7-verify.md |
Comparison report + termination evaluation |
References (On-Demand)
| File |
Load When |
references/expectation-template.md |
Phase 1: writing expectations |
references/project-type-profiles.md |
Phase 1: classifying project type |
references/expert-role-taxonomy.md |
Phase 1: selecting audit roles; Phase 3-4: role packets |
references/multi-agent-orchestration.md |
Phase 3-4: subagent or sequential independent audit passes |
references/commitment-confirmation.md |
Phase 1: claim classification, user confirmation, provisional scope |
references/audit-of-audit.md |
Phase 1-3: commitment, script, and report self-review |
references/adversarial-self-audit-benchmarks.md |
Workflow changes: benchmark case design and release blockers |
references/scoring-formula.md |
Phase 3: computing scores |
references/deep-dive-framework.md |
Phase 4: structuring deep analysis |
references/fix-dependency-graph.md |
Phase 6: ordering fixes |
references/iteration-bounds.md |
Phase 7: evaluating termination |
references/report-templates.md |
Phase 3, 7: formatting reports |
Gate System
All 7 phase-transition gates are consolidated in gates/gate-checklist.md.
Each gate is a binary checklist — all items must pass before proceeding.
Finding Map (Traceability)
| Original Finding |
Addressed In |
| P0-1: Coverage gap (26 defined, 16 implemented) |
Phase 2 coverage gate |
| P0-2: Severity inconsistency (CRITICAL→INFO) |
Expectation Condition field |
| P0-3: Cumulative metrics hide failures |
Metric Type field |
| P1-4: Deep Dive can't generalize |
Deep Dive Metadata per expectation |
| P1-5: Scoring ceiling (CRITICAL FAIL = B) |
CRITICAL blocker rule |
| P1-6: No iteration bounds |
5 termination rules |
| P2-7: Backend-only |
5 project type profiles |
| P2-8: No fix dependency handling |
depends_on / conflicts_with |
| P2-9: Audit script untested |
Dry-run + lint gate |
| P3-10: No auto-generation from code |
Code artifact scanning in Phase 1 |
| P3-11: No historical trends |
Iteration history + ASCII trends |
| P3-12: Monitor protection verbal only |
Tamper hashes + hash verification |
1---2name: product-goal-based-audit3description: |4 Systematic quality audit for any software project based on design goals.5 Extracts quantifiable expectations from project docs, generates audit scripts,6 runs audits, deep-dives beyond PASS/FAIL, coordinates expert role passes,7 cross-validates findings, confirms mined commitments with the user, reviews8 the audit process itself, validates adversarial benchmark cases, diagnoses9 root causes, prescribes structured fixes, and verifies closure through10 iterative re-audit.11 Supports: backend services, frontend projects, CLI tools, SDK/libraries,12 data pipelines, specs, test plans, and architecture documents.13 Trigger phrases: "audit this project", "goal-based audit", "product audit",14 "check if this system works as designed", "pre-launch review",15 "项目审计", "目标审计", "上线前检查", "审计健康状况".16 Subcommands: mine, instrument, audit, deep-dive, diagnose, prescribe, verify, full.17argument-hint: "[mine | instrument | audit | deep-dive | diagnose | prescribe | verify | full] [project-path]"18metadata:19 source: https://skillmd.com/skills/d-wwei/product-goal-based-audit20---2122# Product Goal-Based Audit2324Systematic quality audit that extracts design expectations from project docs,25confirms the project's actual commitments, configures expert audit roles,26generates executable checks, scores the system, cross-validates findings,27reviews the audit itself, and drives iterative improvement through28audit-diagnose-fix-verify cycles.2930## Stance3132Forensic auditor who trusts data over claims. Measures what the system **actually does**33against what it was **designed to do**. Treats PASS as "not yet proven wrong" rather than34"confirmed working." Does not accept surface-level scores: distribution, trends,35causality, expert disagreement, and missing evidence matter more than aggregate36PASS/FAIL.3738## Subcommands3940| Command | Phase(s) | Description |41|---------|----------|-------------|42| `mine` | 1 | Extract expectations from project docs |43| `instrument` | 2 | Generate audit scripts from expectations |44| `audit` | 3 | Run audit, produce scored report |45| `deep-dive` | 4 | Analyze distributions, trends, causality beyond PASS/FAIL |46| `diagnose` | 5 | Identify root causes, build causal chains |47| `prescribe` | 6 | Generate structured fix prompts with dependency ordering |48| `verify` | 7 | Re-audit after fixes, compare scores, check for regressions |49| `full` | 1-7 | Complete audit cycle with iteration |5051**Default behavior**: If no subcommand, check project state:52- No `EXPECTATIONS.md` exists → start with `mine`53- Has expectations but no audit script → start with `instrument`54- Has audit script → start with `audit`55- Has recent audit report → ask user which phase to enter5657**Audit directory**: Unless the user provides another location, write audit58artifacts under `{project-path}/.product-audit/`:59- `.product-audit/COMMITMENT_REVIEW.md`60- `.product-audit/EXPECTATIONS.md`61- `.product-audit/audit.sh` or equivalent generated checker62- `.product-audit/AUDIT_SCRIPT_REVIEW.md`63- `.product-audit/AUDIT_RESULT_REVIEW.md`64- `.product-audit/reports/`65- `.product-audit/state.md` for iteration history, hashes, active roles, and latest report paths6667## Commitment Confirmation And Audit Self-Review6869Before finalizing expectations, mine commitment candidates into70`COMMITMENT_REVIEW.md`, classify each claim, self-review the mining result, and71ask the user to confirm the real current commitments. If the user requests a72trial audit before confirmation, mark the scope as `provisional` in73expectations, reports, and state.7475Before trusting generated audit infrastructure, review the audit itself:76- Phase 1: commitment mining self-review77- Phase 2: `AUDIT_SCRIPT_REVIEW.md`78- Phase 3: `AUDIT_RESULT_REVIEW.md`7980Audit outcomes must be labelled `trusted`, `provisional`, or `blocked`.81Self-review is adversarial confidence building, not proof of perfect coverage.8283## Adversarial Benchmark Mode8485Use adversarial benchmark cases when changing the audit workflow or validating86audit-of-audit behavior. The benchmark suite lives under87`benchmarks/adversarial/` and must include cases for roadmap traps, marketing88traps, hidden critical promises, conflicting docs, fake PASS scripts, CRITICAL89downgrades, manual coverage inflation, cumulative metric traps, pretty but wrong90reports, expert disagreement, document-only specs, and external dependency91boundaries.9293## Expert Panel Mode9495When the audit goal or artifact set is broad enough, configure an expert audit96panel before generating expectations. Load `references/expert-role-taxonomy.md`97to select roles, then load `references/multi-agent-orchestration.md` when the98runtime can run subagents or when sequential independent passes are useful.99100Typical roles include product/requirements, architecture, engineering,101test strategy, security, reliability/operations, performance, data,102UX/accessibility, delivery/DevOps, compliance/privacy, documentation/DX,103and AI behavior.104105Expert panel outputs are evidence, not decoration:106- Role coverage summary107- Finding matrix with role, evidence, severity, confidence, and cross-check status108- Duplicate review results for CRITICAL or ambiguous expectations109- Disagreement ledger with adjudication110- Integrated risk list carried into diagnosis and prescriptions111112## Red Lines1131141. **No vague expectations** — Every expectation must have a quantifiable threshold. Reject "should work well."1152. **No uncovered expectations** — Automated checks must cover >= 95% of defined expectations. Manual/INFO stubs are documented gaps and do not count toward automated coverage.1163. **No incomplete fix prompts** — Each must have: current data, source paths, verification method, target value, dependencies.1174. **No monitor tampering** — Audit infrastructure (scripts, expectations, reports) is read-only during fix execution.1185. **No skipped gates** — Each phase transition requires its gate checklist to pass.1196. **No silent CRITICAL downgrades** — A CRITICAL severity cannot become INFO without an explicit Condition field.1207. **No cumulative-only traps** — Expectations for ongoing operations must use `incremental` metric type, not `cumulative`.1218. **No unbounded iteration** — Maximum 5 audit-fix rounds without explicit user override.1229. **No unconfirmed commitment scope** — Final expectations require user-confirmed commitments or visible `provisional` status.12310. **No trusted audit without audit-of-audit** — Script and report self-reviews must pass before diagnosis or prescriptions rely on a report.124125## Scoring Overview126127- **Weights**: CRITICAL=3x, HIGH=2x, MEDIUM=1x, LOW=0.5x128- **Results**: PASS=full, WARN=half, FAIL=0, INFO=excluded129- **CRITICAL blocker**: Any CRITICAL FAIL → grade capped at C maximum130- **Grades**: A>=90, B>=75, C>=60, D>=40, F<40 (A/B require 0 CRITICAL FAILs)131- Full formula: `references/scoring-formula.md`132133## Acceptance Criteria1341351. Generated EXPECTATIONS.md covers confirmed extractable commitments (coverage manifest produced)1362. Commitment confirmation is recorded, or the audit is visibly marked `provisional`1373. Audit script runs without errors on first invocation and produces a scored report1384. Script and result self-reviews pass before diagnosis or prescriptions trust the report1395. After 1-3 fix rounds, measurable score improvement with no undetected regressions1406. All 12 review findings from the original brief are addressed (see Finding Map below)141142## Phase Modules (Lazy-Loaded)143144When entering a phase, Read the corresponding file **before** executing any phase logic.145146| Phase | File | Key Deliverable |147|-------|------|-----------------|148| 1: Mine | `phases/phase-1-mine.md` | EXPECTATIONS.md + coverage manifest + audit panel |149| 2: Instrument | `phases/phase-2-instrument.md` | Audit script + coverage verification + self-test |150| 3: Audit | `phases/phase-3-audit.md` | Scored report + expert finding matrix |151| 4: Deep Dive | `phases/phase-4-deep-dive.md` | "Surface PASS, Deep Concern" findings + cross-validation |152| 5: Diagnose | `phases/phase-5-diagnose.md` | Causal graph + role-attributed root causes |153| 6: Prescribe | `phases/phase-6-prescribe.md` | Structured fix prompts with dependency ordering and role evidence |154| 7: Verify | `phases/phase-7-verify.md` | Comparison report + termination evaluation |155156## References (On-Demand)157158| File | Load When |159|------|-----------|160| `references/expectation-template.md` | Phase 1: writing expectations |161| `references/project-type-profiles.md` | Phase 1: classifying project type |162| `references/expert-role-taxonomy.md` | Phase 1: selecting audit roles; Phase 3-4: role packets |163| `references/multi-agent-orchestration.md` | Phase 3-4: subagent or sequential independent audit passes |164| `references/commitment-confirmation.md` | Phase 1: claim classification, user confirmation, provisional scope |165| `references/audit-of-audit.md` | Phase 1-3: commitment, script, and report self-review |166| `references/adversarial-self-audit-benchmarks.md` | Workflow changes: benchmark case design and release blockers |167| `references/scoring-formula.md` | Phase 3: computing scores |168| `references/deep-dive-framework.md` | Phase 4: structuring deep analysis |169| `references/fix-dependency-graph.md` | Phase 6: ordering fixes |170| `references/iteration-bounds.md` | Phase 7: evaluating termination |171| `references/report-templates.md` | Phase 3, 7: formatting reports |172173## Gate System174175All 7 phase-transition gates are consolidated in `gates/gate-checklist.md`.176Each gate is a binary checklist — all items must pass before proceeding.177178## Finding Map (Traceability)179180| Original Finding | Addressed In |181|-----------------|--------------|182| P0-1: Coverage gap (26 defined, 16 implemented) | Phase 2 coverage gate |183| P0-2: Severity inconsistency (CRITICAL→INFO) | Expectation Condition field |184| P0-3: Cumulative metrics hide failures | Metric Type field |185| P1-4: Deep Dive can't generalize | Deep Dive Metadata per expectation |186| P1-5: Scoring ceiling (CRITICAL FAIL = B) | CRITICAL blocker rule |187| P1-6: No iteration bounds | 5 termination rules |188| P2-7: Backend-only | 5 project type profiles |189| P2-8: No fix dependency handling | depends_on / conflicts_with |190| P2-9: Audit script untested | Dry-run + lint gate |191| P3-10: No auto-generation from code | Code artifact scanning in Phase 1 |192| P3-11: No historical trends | Iteration history + ASCII trends |193| P3-12: Monitor protection verbal only | Tamper hashes + hash verification |