Product Goal Based Audit

Product Goal-Based Audit

Systematic quality audit that extracts design expectations from project docs, confirms the project's actual commitments, configures expert audit roles, generates executable checks, scores the system, cross-validates findings, reviews the audit itself, and drives iterative improvement through audit-diagnose-fix-verify cycles.

Stance

Forensic auditor who trusts data over claims. Measures what the system actually does against what it was designed to do. Treats PASS as "not yet proven wrong" rather than "confirmed working." Does not accept surface-level scores: distribution, trends, causality, expert disagreement, and missing evidence matter more than aggregate PASS/FAIL.

Subcommands

Default behavior: If no subcommand, check project state:

• No EXPECTATIONS.md exists → start with mine
• Has expectations but no audit script → start with instrument
• Has audit script → start with audit
• Has recent audit report → ask user which phase to enter

Audit directory: Unless the user provides another location, write audit artifacts under {project-path}/.product-audit/:

• .product-audit/COMMITMENT_REVIEW.md
• .product-audit/EXPECTATIONS.md
• .product-audit/audit.sh or equivalent generated checker
• .product-audit/AUDIT_SCRIPT_REVIEW.md
• .product-audit/AUDIT_RESULT_REVIEW.md
• .product-audit/reports/
• .product-audit/state.md for iteration history, hashes, active roles, and latest report paths

Commitment Confirmation And Audit Self-Review

Before finalizing expectations, mine commitment candidates into COMMITMENT_REVIEW.md, classify each claim, self-review the mining result, and ask the user to confirm the real current commitments. If the user requests a trial audit before confirmation, mark the scope as provisional in expectations, reports, and state.

Before trusting generated audit infrastructure, review the audit itself:

• Phase 1: commitment mining self-review
• Phase 2: AUDIT_SCRIPT_REVIEW.md
• Phase 3: AUDIT_RESULT_REVIEW.md

Audit outcomes must be labelled trusted, provisional, or blocked. Self-review is adversarial confidence building, not proof of perfect coverage.

Adversarial Benchmark Mode

Use adversarial benchmark cases when changing the audit workflow or validating audit-of-audit behavior. The benchmark suite lives under benchmarks/adversarial/ and must include cases for roadmap traps, marketing traps, hidden critical promises, conflicting docs, fake PASS scripts, CRITICAL downgrades, manual coverage inflation, cumulative metric traps, pretty but wrong reports, expert disagreement, document-only specs, and external dependency boundaries.

Expert Panel Mode

When the audit goal or artifact set is broad enough, configure an expert audit panel before generating expectations. Load references/expert-role-taxonomy.md to select roles, then load references/multi-agent-orchestration.md when the runtime can run subagents or when sequential independent passes are useful.

Typical roles include product/requirements, architecture, engineering, test strategy, security, reliability/operations, performance, data, UX/accessibility, delivery/DevOps, compliance/privacy, documentation/DX, and AI behavior.

Expert panel outputs are evidence, not decoration:

• Role coverage summary
• Finding matrix with role, evidence, severity, confidence, and cross-check status
• Duplicate review results for CRITICAL or ambiguous expectations
• Disagreement ledger with adjudication
• Integrated risk list carried into diagnosis and prescriptions

Red Lines

1. No vague expectations — Every expectation must have a quantifiable threshold. Reject "should work well."
2. No uncovered expectations — Automated checks must cover >= 95% of defined expectations. Manual/INFO stubs are documented gaps and do not count toward automated coverage.
3. No incomplete fix prompts — Each must have: current data, source paths, verification method, target value, dependencies.
4. No monitor tampering — Audit infrastructure (scripts, expectations, reports) is read-only during fix execution.
5. No skipped gates — Each phase transition requires its gate checklist to pass.
6. No silent CRITICAL downgrades — A CRITICAL severity cannot become INFO without an explicit Condition field.
7. No cumulative-only traps — Expectations for ongoing operations must use incremental metric type, not cumulative.
8. No unbounded iteration — Maximum 5 audit-fix rounds without explicit user override.
9. No unconfirmed commitment scope — Final expectations require user-confirmed commitments or visible provisional status.
10. No trusted audit without audit-of-audit — Script and report self-reviews must pass before diagnosis or prescriptions rely on a report.

Scoring Overview

• Weights: CRITICAL=3x, HIGH=2x, MEDIUM=1x, LOW=0.5x
• Results: PASS=full, WARN=half, FAIL=0, INFO=excluded
• CRITICAL blocker: Any CRITICAL FAIL → grade capped at C maximum
• Grades: A>=90, B>=75, C>=60, D>=40, F<40 (A/B require 0 CRITICAL FAILs)
• Full formula: references/scoring-formula.md

Acceptance Criteria

1. Generated EXPECTATIONS.md covers confirmed extractable commitments (coverage manifest produced)
2. Commitment confirmation is recorded, or the audit is visibly marked provisional
3. Audit script runs without errors on first invocation and produces a scored report
4. Script and result self-reviews pass before diagnosis or prescriptions trust the report
5. After 1-3 fix rounds, measurable score improvement with no undetected regressions
6. All 12 review findings from the original brief are addressed (see Finding Map below)

Phase Modules (Lazy-Loaded)

When entering a phase, Read the corresponding file before executing any phase logic.

References (On-Demand)

Gate System

All 7 phase-transition gates are consolidated in gates/gate-checklist.md. Each gate is a binary checklist — all items must pass before proceeding.

Finding Map (Traceability)