Harness Audit

Harness Audit

Cross-dimensional codebase audit orchestrator — classify the repo's shape, fan out parallel read-only audit agents that compose existing harness skills, adversarially confirm high-severity findings, dedup against the existing issue tracker, and publish grouped thematic tracking issues with checkbox action lists. Optionally hand off to worktree-isolated fix agents, one per theme, gated on human confirmation.

Motivated by repeated manual audits of downstream overlay repos: a full sweep surfaced things no single skill did — a CI security gate silently dead for weeks behind a stale path filter, the highest-blast-radius tools entirely untested, and rename residue that had turned an architecture-enforcement layer into a no-op. The pattern (fan out → verify → dedup → group → optionally fix) proved repeatable and is codified here.

When to Use

• Periodic deep hygiene sweep of a repo you maintain (quarterly, pre-release, taking over a codebase)
• After a large migration, rename, or restructuring, when cross-cutting residue is likely
• When onboarding to an unfamiliar repo and you want a ranked map of what is actually broken
• When individual skills (harness-code-review, harness-security-scan, harness-docs-pipeline) keep finding local issues but nobody has the cross-dimensional picture
• NOT for reviewing a single PR or diff — use harness-code-review
• NOT for a single-concern check (security only, docs only) — invoke that skill directly; the orchestrator's value is corroboration across dimensions
• NOT as an always-on CI gate — this is the on-demand deep sweep; per-PR slices belong to the individual pipelines
• NOT when the working tree is dirty or tests are failing for known in-progress reasons — audit a clean checkpoint, or the noise floor swamps the findings

Flags

Process

Iron Law

Audit phases observe. Only the gated remediation phase changes anything.

Every agent dispatched in Phases 1–5 is read-only: no file writes, no commits, no issue edits, no config changes. An auditor who fixes what it finds has destroyed the evidence and skipped the dedup, grouping, and prioritization that make the audit useful. The only phase that modifies the repo is Phase 6, and it never starts without explicit human confirmation per theme.

Phase 1: CLASSIFY --> Phase 2: FAN-OUT --> Phase 3: VERIFY
                                                      |
                                                      v
Phase 6: REMEDIATE (opt) <-- Phase 5: PUBLISH <-- Phase 4: DEDUP+RANK

Phase 1: CLASSIFY — Repo Shape Gates the Dimensions

1. Classify the repo shape before dispatching anything: app, library, CLI-toolset, skills-overlay, docs-site, or monorepo (which is classified per workspace package). Read the manifest(s), directory layout, and CI workflows to decide.

2. Select dimensions by shape. Graph/entropy/architecture analyzers produce pure noise on a non-app repo — an audit must not run them blindly. Candidate dimensions:

   Dimension	Composes	Skip when
   Structure / arch conformance	harness-dependency-health, enforce-architecture, harness-hotspot-detector	Repo has no source graph (docs/skills overlay)
   Code quality / bugs	harness-code-review (repo-scoped), cleanup-dead-code	No executable source
   Docs drift	detect-doc-drift, validate-context-engineering	Never — every repo has docs to drift
   CI / workflow hygiene	Direct inspection of .github/workflows/ (or a workflow-audit skill if one is present)	No CI configured (that IS a finding)
   Security posture	harness-security-scan, security-craft, harness-supply-chain-audit	Never
   Test depth	harness-test-advisor, test-craft (JS/TS) or the repo's test-intelligence skills (other languages)	No test surface exists (that IS a finding)
   UX / onboarding friction	Fresh-eyes walkthrough of README, setup docs, first-run experience	Internal single-maintainer tooling

3. Scale agent count to repo size. Small repo (under roughly 5K LOC, or overlay-shaped): merge related dimensions into 3 agents. Medium: 4–5. Large app or monorepo: full roster up to --max-agents. Record the roster and the skip rationale for every excluded dimension.

4. Build the exclusion list. Generated files, lockfiles, vendored code, ledger/baseline files (coverage baselines, security ledgers, snapshot fixtures), and build output are out of scope for every dimension. Findings inside them are almost always false positives against machine-written content. Record the glob list now and pass it to every agent.

Phase 2: FAN-OUT — Parallel Read-Only Audit Agents

1. Dispatch one agent per dimension, in parallel. Each agent receives: the dimension charter, the exclusion list, the repo-shape classification, and the instruction to compose the existing skill for its dimension rather than re-derive its logic. The security agent runs harness-security-scan and reads its output; it does not reinvent secret-detection regexes.

2. Search prose, not just code. Each agent greps PR descriptions, issue text, commit messages, and docs — not only diffs and source. Dead CI gates, contradicted decisions, and abandoned migrations are usually visible in prose long before they are visible in code (gh pr list --search, gh issue list --search, git log --grep).

3. Require evidence per finding. Every finding returns as:

   AuditFinding {
     id, dimension, severity: HIGH | MEDIUM | LOW,
     file/lineRange (or workflow/issue/PR reference),
     evidence,           // the exact observation, quotable
     reproduction,       // how a second agent can confirm it
     suggestedAction,    // one checkbox-sized action
     ownerRepo,          // when the audit spans repos, which repo owns the fix
   }
   

   A finding without reproduction steps is an opinion, not a finding. Agents must not write files, post comments, or open issues — findings come back in-band.

4. Collect all agents before proceeding. A dimension agent that errors is retried once; if it fails again, record the dimension as UNAUDITED in the final report rather than silently narrowing scope.

Phase 3: VERIFY — Adversarial Confirmation of HIGH Findings

1. Verify, don't trust. For every HIGH finding, dispatch a second agent whose charter is to disprove it: reproduce the evidence independently, check whether the "dead gate" actually fires on some other path, whether the "untested module" is covered by an integration suite the first agent missed, whether the "secret" is a documented placeholder.

2. Each HIGH finding exits this phase as confirmed (second agent reproduced it), downgraded (real but overstated), or discarded (not reproducible — record why, so the false-positive pattern feeds back into the dimension charter).

3. Spot-check MEDIUMs. Sample at least 20% of MEDIUM findings with the same adversarial pass. If more than a third of the sample fails confirmation, verify all MEDIUMs before proceeding — the dimension agent was hallucination-prone.

Phase 4: DEDUP + RANK — One Truth Per Problem

1. Merge cross-dimension duplicates. The same root cause frequently surfaces in multiple dimensions (a stale path filter appears as CI hygiene AND security posture). Merge into one finding, keep both evidence trails, and mark it corroborated — a finding two dimensions hit independently is high-confidence and ranks above its severity peers.

2. Cross-check the issue tracker. Run gh issue list --state open (and --state closed --search for recently closed) and match findings against existing issues by symptom, file, and keyword — search issue text, not just titles. A finding already tracked is annotated with the issue reference and excluded from filing; a finding matching a recently closed issue is flagged as a possible regression instead.

3. Severity-rank the survivors: confirmed-corroborated HIGH first, then confirmed HIGH, then MEDIUM by blast radius (use harness impact-preview / hotspot data where available), then LOW. LOW findings that share a theme with higher findings ride along in that theme; orphan LOWs go into a single hygiene bucket.

Phase 5: PUBLISH — Grouped Thematic Tracking Issues

1. Group findings into themes, not per-finding issues. A theme is a coherent unit of remediation work (e.g. "CI workflow hygiene", "test coverage for high-blast-radius tools", "rename residue"). Target 3–7 themes; 30 single-finding issues is spam, one mega-issue is unactionable.

2. File one tracking issue per theme (skip when --report-only or --ci; emit the ranked report instead). Each issue contains:

   • A one-paragraph theme summary with severity and corroboration
   • A checkbox list (- [ ]) of concrete actions, one per finding, each with a file/line or workflow reference and an evidence link
   • Owner-repo attribution when findings span repos: file the issue in the repo that owns the fix, or clearly label per-checkbox which repo owns each action
   • Cross-references to the existing issues found in Phase 4 rather than restating them

3. Emit the audit report: dimension roster (including skipped and UNAUDITED dimensions with rationale), finding counts by severity before/after verification, dedup statistics, and links to the filed issues.

Phase 6: REMEDIATE (Optional) — Gated Fix Fan-Out

1. Only with --fix, and only per theme after explicit human confirmation. Present the themes; the human picks which (if any) to dispatch. No confirmation, no remediation — silence is a "no".

2. One worktree-isolated fix agent per confirmed theme. Each agent works on a branch in its own worktree, addresses its theme's checkboxes, runs the repo's full gates, and opens a PR that references (and where complete, closes) its tracking issue.

3. Fix agents are ordinary contributors: conventional commits, no gate bypasses, PR review flow. The audit orchestrator does not merge.

Harness Integration

• harness skill run harness-audit — Run the full audit pipeline.
• harness skill run harness-code-review — Composed by the code-quality dimension.
• harness check-security / harness skill run security-craft — Composed by the security dimension.
• harness skill run detect-doc-drift — Composed by the docs dimension.
• harness check-deps / harness skill run harness-dependency-health — Composed by the structure dimension (app-shaped repos only).
• harness skill run harness-test-advisor — Composed by the test-depth dimension.
• harness impact-preview — Blast-radius input for severity ranking in Phase 4.

Success Criteria

• Repo shape classified and every skipped dimension has a recorded rationale
• All dispatched audit agents completed (or are listed as UNAUDITED — never silently dropped)
• Zero HIGH findings published without a second-agent confirmation verdict
• No published finding references a generated, vendored, or ledger file from the exclusion list
• No filed issue duplicates an existing open issue; regressions of closed issues are labeled as such
• Themes number between 3 and 7 (plus at most one hygiene bucket), each with a checkbox action list and owner-repo attribution where relevant
• Nothing in the repo changed unless Phase 6 ran with explicit per-theme confirmation

Gates

• Read-only until Phase 6. Any file write, commit, issue edit, or config change during Phases 1–5 = audit invalid. Stop, revert, restart the affected phase.
• No fan-out without classification. Dispatching dimension agents before the repo-shape roster exists = noise. Phase 1 output is a prerequisite for Phase 2.
• No publishing unverified HIGHs. A HIGH finding that skipped Phase 3 must not appear in any issue or report. If verification cannot run, downgrade to MEDIUM and say why.
• No filing without the tracker cross-check. If gh issue list is unavailable (no network, no auth), fall back to --report-only — never file blind.
• No remediation without explicit confirmation. --fix enables the offer; each theme's fix agent starts only on an explicit human yes for that theme.

Escalation

• When two dimensions return contradictory findings (e.g. "this module is dead code" vs "this module is the highest-blast-radius untested path"): do not average them. Report both with evidence and let the human adjudicate — contradiction usually means the repo classification or exclusion list is wrong.
• When the tracker cross-check finds a recently closed issue matching a live finding: flag as a possible regression, link the closing PR, and escalate to the human before filing — re-filing a fixed-and-regressed issue as "new" loses the history.
• When more than a third of sampled findings fail adversarial verification: the dimension agent is unreliable. Report the dimension as UNAUDITED with the failure pattern rather than publishing its remainder unverified.
• When the audit spans repos and fix ownership is ambiguous: file nothing across the boundary; present the ownership question to the human with the evidence from both repos.

Rationalizations to Reject

Examples

Example: Auditing a medium-sized CLI toolset

$ harness skill run harness-audit

Phase 1: CLASSIFY
  Shape: CLI-toolset (bin entries, no served app). ~22K LOC TypeScript.
  Roster (5 agents): code-quality, security, docs, ci-hygiene, test-depth.
  Skipped: structure/arch (no layered app graph — noise on toolsets),
           ux-onboarding (merged into docs — single-audience internal tool).
  Exclusions: dist/**, pnpm-lock.yaml, coverage-baselines.json, **/__snapshots__/**

Phase 2: FAN-OUT (parallel)
  code-quality  -> 9 findings (composed harness-code-review, repo scope)
  security      -> 3 findings (composed harness-security-scan + supply-chain-audit)
  docs          -> 6 findings (composed detect-doc-drift)
  ci-hygiene    -> 4 findings (inspected .github/workflows + gh run list)
  test-depth    -> 5 findings (composed harness-test-advisor + harness impact-preview)

Phase 3: VERIFY
  4 HIGH findings -> 3 confirmed, 1 discarded (the "hardcoded token" is a
  documented placeholder; charter updated).

Phase 4: DEDUP+RANK
  26 -> 18 findings. 1 corroborated: security + ci-hygiene independently
  found the scan workflow's path filter references a directory renamed in
  PR #212 — the gate has not executed in 6 weeks.
  Tracker cross-check: 2 findings already open (#88, #103) — annotated, not re-filed.
  1 finding matches closed #71 — flagged as regression, escalated.

Phase 5: PUBLISH
  4 theme issues filed:
    #240 CI gate integrity (1 corroborated HIGH + 2 MEDIUM, 4 checkboxes)
    #241 Test depth on high-blast-radius tools (1 HIGH + 3 MEDIUM)
    #242 Docs drift after the v3 rename (5 findings)
    #243 Hygiene bucket (4 LOW)
  Report: 5/5 dimensions audited, 0 UNAUDITED.

Phase 6: skipped (--fix not passed).

Example: Rejecting an in-audit fix

While auditing, the ci-hygiene agent proposes: "the path filter fix is a one-line change, I applied it." This violates the Iron Law — the change is reverted, the finding is recorded with the proposed one-liner as its suggestedAction, and it later lands via the Phase 6 fix agent's PR against theme issue #240, where the fix also gains the regression test the drive-by one-liner would have skipped.