# Product Goal Based Audit

> Systematic quality audit for any software project based on design goals. Extracts quantifiable expectations from project docs, generates audit scripts, runs audits, deep-dives beyond PASS/FAIL, coordinates expert role passes, cross-validates findings, confirms mined commitments with the user, reviews the audit process itself, validates adversarial benchmark cases, diagnoses root causes, prescribes structured fixes, and verifies closure through iterative re-audit. Supports: backend services, frontend projects, CLI tools, SDK/libraries, data pipelines, specs, test plans, and architecture documents. Trigger phrases: "audit this project", "goal-based audit", "product audit", "check if this system works as designed", "pre-launch review", "项目审计", "目标审计", "上线前检查", "审计健康状况". Subcommands: mine, instrument, audit, deep-dive, diagnose, prescribe, verify, full.

- Skill: `d-wwei/product-goal-based-audit` (Agent Skill, multi-file: 7 files)
- Install (CLI): `npx skillmds@latest add d-wwei/product-goal-based-audit`
- Raw SKILL.md: https://api.skillmd.com/api/skills/d-wwei/product-goal-based-audit/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: d-wwei (https://skillmd.com/u/d-wwei)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/d-wwei/product-goal-based-audit

---


# Product Goal-Based Audit

Systematic quality audit that extracts design expectations from project docs,
confirms the project's actual commitments, configures expert audit roles,
generates executable checks, scores the system, cross-validates findings,
reviews the audit itself, and drives iterative improvement through
audit-diagnose-fix-verify cycles.

## Stance

Forensic auditor who trusts data over claims. Measures what the system **actually does**
against what it was **designed to do**. Treats PASS as "not yet proven wrong" rather than
"confirmed working." Does not accept surface-level scores: distribution, trends,
causality, expert disagreement, and missing evidence matter more than aggregate
PASS/FAIL.

## Subcommands

| Command | Phase(s) | Description |
|---------|----------|-------------|
| `mine` | 1 | Extract expectations from project docs |
| `instrument` | 2 | Generate audit scripts from expectations |
| `audit` | 3 | Run audit, produce scored report |
| `deep-dive` | 4 | Analyze distributions, trends, causality beyond PASS/FAIL |
| `diagnose` | 5 | Identify root causes, build causal chains |
| `prescribe` | 6 | Generate structured fix prompts with dependency ordering |
| `verify` | 7 | Re-audit after fixes, compare scores, check for regressions |
| `full` | 1-7 | Complete audit cycle with iteration |

**Default behavior**: If no subcommand, check project state:
- No `EXPECTATIONS.md` exists → start with `mine`
- Has expectations but no audit script → start with `instrument`
- Has audit script → start with `audit`
- Has recent audit report → ask user which phase to enter

**Audit directory**: Unless the user provides another location, write audit
artifacts under `{project-path}/.product-audit/`:
- `.product-audit/COMMITMENT_REVIEW.md`
- `.product-audit/EXPECTATIONS.md`
- `.product-audit/audit.sh` or equivalent generated checker
- `.product-audit/AUDIT_SCRIPT_REVIEW.md`
- `.product-audit/AUDIT_RESULT_REVIEW.md`
- `.product-audit/reports/`
- `.product-audit/state.md` for iteration history, hashes, active roles, and latest report paths

## Commitment Confirmation And Audit Self-Review

Before finalizing expectations, mine commitment candidates into
`COMMITMENT_REVIEW.md`, classify each claim, self-review the mining result, and
ask the user to confirm the real current commitments. If the user requests a
trial audit before confirmation, mark the scope as `provisional` in
expectations, reports, and state.

Before trusting generated audit infrastructure, review the audit itself:
- Phase 1: commitment mining self-review
- Phase 2: `AUDIT_SCRIPT_REVIEW.md`
- Phase 3: `AUDIT_RESULT_REVIEW.md`

Audit outcomes must be labelled `trusted`, `provisional`, or `blocked`.
Self-review is adversarial confidence building, not proof of perfect coverage.

## Adversarial Benchmark Mode

Use adversarial benchmark cases when changing the audit workflow or validating
audit-of-audit behavior. The benchmark suite lives under
`benchmarks/adversarial/` and must include cases for roadmap traps, marketing
traps, hidden critical promises, conflicting docs, fake PASS scripts, CRITICAL
downgrades, manual coverage inflation, cumulative metric traps, pretty but wrong
reports, expert disagreement, document-only specs, and external dependency
boundaries.

## Expert Panel Mode

When the audit goal or artifact set is broad enough, configure an expert audit
panel before generating expectations. Load `references/expert-role-taxonomy.md`
to select roles, then load `references/multi-agent-orchestration.md` when the
runtime can run subagents or when sequential independent passes are useful.

Typical roles include product/requirements, architecture, engineering,
test strategy, security, reliability/operations, performance, data,
UX/accessibility, delivery/DevOps, compliance/privacy, documentation/DX,
and AI behavior.

Expert panel outputs are evidence, not decoration:
- Role coverage summary
- Finding matrix with role, evidence, severity, confidence, and cross-check status
- Duplicate review results for CRITICAL or ambiguous expectations
- Disagreement ledger with adjudication
- Integrated risk list carried into diagnosis and prescriptions

## Red Lines

1. **No vague expectations** — Every expectation must have a quantifiable threshold. Reject "should work well."
2. **No uncovered expectations** — Automated checks must cover >= 95% of defined expectations. Manual/INFO stubs are documented gaps and do not count toward automated coverage.
3. **No incomplete fix prompts** — Each must have: current data, source paths, verification method, target value, dependencies.
4. **No monitor tampering** — Audit infrastructure (scripts, expectations, reports) is read-only during fix execution.
5. **No skipped gates** — Each phase transition requires its gate checklist to pass.
6. **No silent CRITICAL downgrades** — A CRITICAL severity cannot become INFO without an explicit Condition field.
7. **No cumulative-only traps** — Expectations for ongoing operations must use `incremental` metric type, not `cumulative`.
8. **No unbounded iteration** — Maximum 5 audit-fix rounds without explicit user override.
9. **No unconfirmed commitment scope** — Final expectations require user-confirmed commitments or visible `provisional` status.
10. **No trusted audit without audit-of-audit** — Script and report self-reviews must pass before diagnosis or prescriptions rely on a report.

## Scoring Overview

- **Weights**: CRITICAL=3x, HIGH=2x, MEDIUM=1x, LOW=0.5x
- **Results**: PASS=full, WARN=half, FAIL=0, INFO=excluded
- **CRITICAL blocker**: Any CRITICAL FAIL → grade capped at C maximum
- **Grades**: A>=90, B>=75, C>=60, D>=40, F<40 (A/B require 0 CRITICAL FAILs)
- Full formula: `references/scoring-formula.md`

## Acceptance Criteria

1. Generated EXPECTATIONS.md covers confirmed extractable commitments (coverage manifest produced)
2. Commitment confirmation is recorded, or the audit is visibly marked `provisional`
3. Audit script runs without errors on first invocation and produces a scored report
4. Script and result self-reviews pass before diagnosis or prescriptions trust the report
5. After 1-3 fix rounds, measurable score improvement with no undetected regressions
6. All 12 review findings from the original brief are addressed (see Finding Map below)

## Phase Modules (Lazy-Loaded)

When entering a phase, Read the corresponding file **before** executing any phase logic.

| Phase | File | Key Deliverable |
|-------|------|-----------------|
| 1: Mine | `phases/phase-1-mine.md` | EXPECTATIONS.md + coverage manifest + audit panel |
| 2: Instrument | `phases/phase-2-instrument.md` | Audit script + coverage verification + self-test |
| 3: Audit | `phases/phase-3-audit.md` | Scored report + expert finding matrix |
| 4: Deep Dive | `phases/phase-4-deep-dive.md` | "Surface PASS, Deep Concern" findings + cross-validation |
| 5: Diagnose | `phases/phase-5-diagnose.md` | Causal graph + role-attributed root causes |
| 6: Prescribe | `phases/phase-6-prescribe.md` | Structured fix prompts with dependency ordering and role evidence |
| 7: Verify | `phases/phase-7-verify.md` | Comparison report + termination evaluation |

## References (On-Demand)

| File | Load When |
|------|-----------|
| `references/expectation-template.md` | Phase 1: writing expectations |
| `references/project-type-profiles.md` | Phase 1: classifying project type |
| `references/expert-role-taxonomy.md` | Phase 1: selecting audit roles; Phase 3-4: role packets |
| `references/multi-agent-orchestration.md` | Phase 3-4: subagent or sequential independent audit passes |
| `references/commitment-confirmation.md` | Phase 1: claim classification, user confirmation, provisional scope |
| `references/audit-of-audit.md` | Phase 1-3: commitment, script, and report self-review |
| `references/adversarial-self-audit-benchmarks.md` | Workflow changes: benchmark case design and release blockers |
| `references/scoring-formula.md` | Phase 3: computing scores |
| `references/deep-dive-framework.md` | Phase 4: structuring deep analysis |
| `references/fix-dependency-graph.md` | Phase 6: ordering fixes |
| `references/iteration-bounds.md` | Phase 7: evaluating termination |
| `references/report-templates.md` | Phase 3, 7: formatting reports |

## Gate System

All 7 phase-transition gates are consolidated in `gates/gate-checklist.md`.
Each gate is a binary checklist — all items must pass before proceeding.

## Finding Map (Traceability)

| Original Finding | Addressed In |
|-----------------|--------------|
| P0-1: Coverage gap (26 defined, 16 implemented) | Phase 2 coverage gate |
| P0-2: Severity inconsistency (CRITICAL→INFO) | Expectation Condition field |
| P0-3: Cumulative metrics hide failures | Metric Type field |
| P1-4: Deep Dive can't generalize | Deep Dive Metadata per expectation |
| P1-5: Scoring ceiling (CRITICAL FAIL = B) | CRITICAL blocker rule |
| P1-6: No iteration bounds | 5 termination rules |
| P2-7: Backend-only | 5 project type profiles |
| P2-8: No fix dependency handling | depends_on / conflicts_with |
| P2-9: Audit script untested | Dry-run + lint gate |
| P3-10: No auto-generation from code | Code artifact scanning in Phase 1 |
| P3-11: No historical trends | Iteration history + ASCII trends |
| P3-12: Monitor protection verbal only | Tamper hashes + hash verification |