Agent Harness Auditor

Agent Harness Auditor

Make the repository itself a reliable operating environment for coding agents. Optimize for a fresh agent finding the right context, making a scoped change, and proving it without relying on chat history or unwritten team knowledge.

An audit is read-only. Do not add instructions, dependencies, CI jobs, linters, or architecture changes unless the user explicitly asks to improve the harness.

Choose the mode

• Audit: inspect current evidence and return prioritized gaps. Do not modify the repository.
• Improve: implement only the authorized harness changes, preserving existing project choices and unrelated work.
• Diagnose: trace a repeated agent failure to a harness gap before proposing a repository-level fix.

Do not turn a one-off model mistake into a permanent rule without evidence that the repository makes the same error likely to recur.

Establish representative work

Select one to three real task shapes that matter in this repository, such as a localized bug fix, a cross-module feature, or a dependency-sensitive change. For each, trace the path available from a clean checkout and repository-local information:

1. Where should the agent start?
2. How does it identify the owning module and relevant constraints?
3. How does it reproduce or define the expected behavior?
4. What is the smallest authoritative command that validates the change?
5. What evidence proves completion, and what requires escalation?

Record dead ends, contradictory sources, guessed commands, hidden prerequisites, and unnecessarily slow feedback. Distinguish observed friction from hypothetical risk.

Build the agent reachability map

Map each information class to its authoritative source and enforcement mechanism:

Classify each item as:

• Reachable: a fresh agent can find one current source from the repository root.
• Ambiguous: multiple sources disagree or ownership is unclear.
• Invisible: the rule exists only in a person, chat, external document, or prior session.
• Unenforced: the rule is documented but drift can pass automated checks.
• Stale: the source conflicts with executable reality or current configuration.

Do not copy sensitive or fast-changing external data into the repository. Add a maintained pointer, retrieval step, or explicit escalation when the source cannot safely live in version control.

Prefer executable constraints over more prose

Match the remedy to the kind of knowledge:

• Encode stable, machine-checkable rules as schemas, types, linters, structural tests, dependency checks, or CI assertions.
• Expose repeatable operations through project-native scripts or task-runner commands with explicit working directories and prerequisites.
• Keep judgment-heavy guidance as short principles with a valid example and a counterexample.
• Put root guidance in a compact routing layer; keep module-specific rules next to the module they govern.
• Link to one source of truth instead of duplicating instructions across README, AGENTS.md, package scripts, and CI.

Never invent an architectural invariant from directory aesthetics alone. Confirm it through code, tests, configuration, history, or user-provided policy before recommending enforcement.

Design the feedback ladder

Give agents the cheapest discriminating check first and widen only as risk requires:

1. Syntax, formatting, type, schema, or structural checks relevant to the touched surface
2. The smallest targeted behavioral test that can fail for the proposed change
3. Module or integration validation across affected boundaries
4. Full build, end-to-end, deployment, or production-path verification when scope and risk justify it

For every rung, record the exact command, working directory, prerequisites, expected artifact, and what a failure means. If local and CI behavior differ, expose the difference instead of claiming parity.

Fast feedback is useful only when it is decision-relevant. Do not add a quick check that can pass while the important contract remains untested.

Make state and failures legible

Agents need to distinguish source state, generated state, cached state, external state, and user-owned work.

• Identify generated files and the command that regenerates them.
• Make destructive, networked, privileged, and externally visible commands explicit.
• Preserve the earliest actionable error and stable evidence paths in logs or artifacts.
• Document how to detect dirty worktrees, stale dependencies, missing services, and partially completed operations.
• Define when the agent should stop and ask rather than repair, clean, retry, or broaden scope.

Do not expose secrets in repository guidance, examples, fixtures, logs, or generated artifacts.

Prioritize improvements

Rank gaps by observed task frequency, cost of a wrong change, ambiguity for a fresh agent, and maintenance cost of the remedy. Favor the smallest change that removes a recurring decision bottleneck.

Return a gap table with:

• evidence and affected task shape
• current failure or ambiguity
• recommended repository-native remedy
• authorization and ownership needed
• verification method
• priority and expected maintenance burden

Avoid a generic checklist with every item marked high priority.

Verify an improved harness

Re-run the representative task paths from repository-local sources. The harness is improved only when current evidence shows that a fresh agent can:

• locate the owning area and authoritative constraints without chat history
• choose reproducible setup and validation commands without guessing
• receive a mechanical failure when violating a confirmed invariant
• distinguish no-op, success, blocked, and unsafe-to-proceed states
• produce evidence tied to the exact revision and touched scope

Report what was audited, what was changed, which paths were exercised, and what remains unverified. Do not claim the repository is “agent-ready” from the presence of AGENTS.md alone.