Harness Engineering Audit

Harness Engineering Audit

Use this skill to turn a repository into a production-ready, agent-legible engineering environment optimized for Codex, OMX, and long-running AI-assisted development.

This skill is audit-first, AGENTS-first, and low-risk auto-approved. It may generate reports and planning artifacts before edits, then low-risk recommendations from the audit are approved for follow-up execution without another user approval. Medium/high-risk changes still require explicit approval.

Core operating model

1. Inspect the repo and collect evidence.
2. Score the repo against the harness-engineering rubric.
3. Produce a strict report with low/medium/high-risk recommendations.
4. Generate AGENTS.md priority and OMX handoff artifacts.
5. Auto-approve low-risk fixes, with root AGENTS.md as P0 when it is missing, oversized, stale, or lacks docs/validation pointers.
6. Classify lifecycle as greenfield-bootstrap, brownfield-cleanup, or mature-audit.
7. Use OMX planning/execution or explicit setup modes for auto-approved low-risk fixes; defer medium/high-risk changes until explicit approval.

The default report location is:

.codex/reports/harness-engineering-audit/

Each new run replaces the previous generated report directory at that path to avoid stale audit confusion. The included scripts only overwrite the known report directory or an explicitly provided output directory that is marked as a harness-engineering audit report directory.

When to use

Use this skill when a user asks to audit or improve:

• harness engineering
• agentic development readiness
• Codex readiness
• OMX workflow quality
• AGENTS.md scope and quality
• .codex/config.toml
• MCP configuration
• skills
• hooks and rules
• subagent/team workflow
• docs authority
• validation commands
• command registries
• generated artifact policy
• stack/tooling upgrade recommendations
• scaffolding/legacy entropy
• repo legibility for AI agents
• production-readiness for agent-first development
• docs-to-production app readiness
• visual product spec packs from generated PNG/SVG targets, UI packs, stories, token packs, and component contracts
• frontend parity and product usability harnesses
• runtime evidence / local product done gates
• prompt-injection and instruction-trust readiness

Authoritative principles

Treat OpenAI/Codex guidance as normative:

• AGENTS.md should be a concise map, not an encyclopedia.
• Repo-local docs should be the system of record.
• Validation should be executable and evidence-backed.
• Skills should encode repeatable workflows.
• Domain vocabulary should be discoverable and progressively disclosed, so agents use project language consistently without bloating hot-path instructions.
• Docs should be gardened as living knowledge: raw sources, synthesized docs, indexes, logs, and health checks need explicit boundaries and maintenance workflows.
• MCP should solve real external/context access problems.
• Hooks/rules should be deterministic, scoped, and safe.
• Subagents/team workflows should have clear delegation, path scope, and review boundaries.
• Generated artifacts and scaffolding require lifecycle policies.
• Generated visual targets are design-intent evidence, not executable specifications; every priority target needs an adjacent screen contract.
• Implied behavior must be classified as explicit, inferred, or unresolved; unresolved and sensitive inferred behavior stays approval-gated.
• Production app readiness requires traceability from user stories to visual targets, screen contracts, routes, APIs, components, tests, screenshots, accessibility evidence, and production deltas.

Use Claude/Cursor/Karpathy/third-party resources only as comparison material, not as authority over Codex behavior. The vocabulary/domain-language audit surface is informed by Matt Pocock's Dictionary of AI Coding as comparison material: shared terms, handoff artifacts, progressive disclosure, and the distinction between deterministic checks and review are useful harness concepts, but this skill should not copy that repo's prose. The doc-gardening audit surface is informed by Andrej Karpathy's LLM Wiki gist as comparison material: persistent markdown knowledge bases, source/wiki/schema layering, ingest/query/lint workflows, indexes/logs, and wiki health checks are useful harness concepts, but this skill should not copy that gist's prose.

Required evidence to inspect

Always inventory these surfaces when present:

• root AGENTS.md
• nested AGENTS.md and AGENTS.override.md
• .codex/config.toml
• .codex/hooks.json
• .codex/rules/**
• .codex/prompts/**
• .agents/skills/**
• .omx/**
• .github/workflows/**
• root README and docs indexes
• docs/**
• canonical product docs such as docs/product/**, docs/stories/**, docs/screens/**, docs/design/**, docs/architecture/**, docs/contracts/**, and docs/quality/**
• generated/imported UI targets such as docs/screens/**/target*.png, docs/screens/**/target*.svg, docs/ui/targets/**, and equivalent Figma/design-export packs
• screen and behavior contracts such as screen.contract.yaml, interactions.yaml, motion.yaml, data-requirements.yaml, route maps, state machines, acceptance matrices, and traceability matrices
• design token and component packs such as tokens.json, docs/design/tokens/**, docs/design/components/**, Style Dictionary/Tokens Studio config, component recipes, and Storybook stories
• visual QA surfaces such as .storybook/**, *.stories.*, playwright.config.*, screenshot baselines, visual-regression policy, accessibility policy, and viewport/theme/reduced-motion matrices
• glossary / terminology / domain-language docs such as docs/GLOSSARY.md, docs/TERMINOLOGY.md, internal/CONTEXT.md, internal/domain.md, and dictionary/**
• ADR / decision-record surfaces such as docs/adr/**, docs/ADR/**, and internal/adr/**
• doc gardening / wiki / knowledge-base surfaces such as raw sources, generated/synthesized docs, index.md, log.md, ingest/query/lint workflows, and stale/contradiction/orphan/broken-link checks
• validation scripts and package scripts
• test/build/lint/typecheck/smoke commands
• generated artifact directories
• scaffold/legacy/stage/demo/preview markers
• cross-agent files: CLAUDE.md, .claude/**, .cursor/**, .windsurf/**, .cline/**, .gemini/**, .github/copilot-instructions.md

Docs-to-production harness audit

The audit now includes a stack-neutral Docs-to-Production App Readiness family. It detects repository capabilities and evaluates whether the repo exposes the harness layers needed for Codex/OMX to build a locally usable product from docs and design targets:

• Visual Product Spec Pack
• Visual Target Contracts
• User Story & Acceptance Pack
• Frontend Parity System
• Design Token & Component Pack
• Component Workbench / Storybook Harness
• Visual Regression Gate
• Product Usability Contract
• Runtime Evidence Harness
• Coverage & Traceability Harness
• Local Provider Simulator Harness
• Brownfield State Inventory Harness
• State Machine Harness
• Negative Path Harness
• Role Walkthrough Harness
• Runtime Observability Harness
• Local Product Done Gate
• Repo-scoped Codex skills
• Instruction Trust / Prompt Injection Audit

For UI-led work, the skill applies these tenets:

• a generated screenshot/mockup is an input artifact, not the contract;
• screen.contract.yaml or an equivalent artifact must state route, viewport, theme, visual state, interactions, data/API assumptions, motion, accessibility, and approval-gated inferences;
• the repo should keep a canonical docs set for product intent, user stories, screens, design tokens/components, architecture, contracts, and quality gates;
• a traceability matrix should connect each priority story to visual targets, screen contracts, routes, components, APIs, fixtures, tests, and evidence;
• visual completion should be proven through a workbench/Storybook surface, E2E/runtime evidence, accessibility checks, and human-approved visual baselines rather than raw prompt assertions.

The audit writes pre-mortem and readiness artifacts under .codex/reports/harness-engineering-audit/, including premortem.md, failure-mode-register.yaml, harness-gap-matrix.yaml, docs-to-production-readiness.md, capability-adapter-report.yaml, instruction-trust-report.md, and reusable OMX prompt files under recommended-omx-prompts/.

product-harness-setup can create low-risk harness templates and repo-scoped skill templates for visual-target ingest, design-token/component contracts, frontend parity, product usability, runtime evidence, traceability, visual QA, and final review. It must not implement application features, enable live providers, mutate security-sensitive configuration, or use real data.

Run the audit script

Before running the audit in an interactive session, prompt the user for the audit level and proceed with their selection. Offer these choices:

The Python script also prompts for this level when run from an interactive TTY without --mode. Non-interactive runs default to minimal/audit so automation remains stable.

From the repo root after installing this skill, run the script from the installed location.

Project-scoped install:

python3 .agents/skills/harness-engineering-audit/scripts/run_audit.py .

User-scoped install:

python3 ~/.codex/skills/harness-engineering-audit/scripts/run_audit.py .

The script writes:

.codex/reports/harness-engineering-audit/
  inventory.json
  scorecard.json
  stack-inventory.json
  tool-inventory.json
  upgrade-recommendations.json
  upgrade-recommendations.md
  web-verification-queue.json
  source-trust-policy.md
  update-status.json
  report.md
  findings.md
  recommended-fixes.md
  agents-priority.md
  omx-handoff.md
  next-step.md
  next-step.json
  docs-to-production-readiness.md
  premortem.md
  premortem.json
  failure-mode-register.yaml
  harness-gap-matrix.yaml
  capability-adapter-report.yaml
  instruction-trust-report.md
  recommended-omx-prompts/
  prompts/
    deep-interview.md
    ralplan.md
    team.md
    ralph.md
    symphony-adoption.md
    tool-upgrade-ralplan.md

Recommendation defaults are intentionally conservative:

• recommend tools: true
• web verification requested: true
• human approval gate: required
• install/config mutation: false
• write reports: true

Local Python scripts cannot browse. They therefore write a web verification queue and keep web_verified: false until a browsing-capable human/agent records evidence. Generated install/config/validation/rollback commands are inert handoff text and must not be executed without explicit approval.

Primary artifacts now include a “What happened / What did not happen / What needs approval” section plus machine-readable mode_summary, approval_state, tool_recommendation_state, and current_step_explanation fields in next-step.json.

Use a custom output directory only when needed:

python3 ~/.codex/skills/harness-engineering-audit/scripts/run_audit.py . --out /tmp/harness-engineering-audit

Execution modes

The default prompt choice remains report-only. Use minimal or audit explicitly when you want to skip the prompt while preserving report-only behavior:

python3 ~/.codex/skills/harness-engineering-audit/scripts/run_audit.py . --mode minimal
python3 ~/.codex/skills/harness-engineering-audit/scripts/run_audit.py . --mode audit

Additional explicit modes create only low-risk harness artifacts with provenance markers and setup-rollback-manifest.json:

AUDIT_SCRIPT=~/.codex/skills/harness-engineering-audit/scripts/run_audit.py
# For a project-scoped install, use:
# AUDIT_SCRIPT=.agents/skills/harness-engineering-audit/scripts/run_audit.py

python3 "$AUDIT_SCRIPT" . --mode safe-setup
python3 "$AUDIT_SCRIPT" . --mode force-ideal-harness
python3 "$AUDIT_SCRIPT" . --mode symphony-repo-local
python3 "$AUDIT_SCRIPT" . --mode symphony-live-handoff
python3 "$AUDIT_SCRIPT" . --mode full-orchestration
python3 "$AUDIT_SCRIPT" . --mode product-harness-setup

--force keeps its existing output-overwrite meaning; it is not the force-ideal-harness mode.

Mode boundaries:

• audit: reports only, no target source mutation.
• safe-setup: creates missing low-risk docs/templates only.
• force-ideal-harness: explicit stronger low-risk consolidation; no deletes or CI/config/hooks/security changes.
• symphony-repo-local: repo-local Symphony contracts/templates and inert handoff text only.
• symphony-live-handoff: approval-gated handoff text only; no install/config mutation.
• full-orchestration: explicit opt-in for lane-pack orchestration contracts and project custom-agent TOML. It is never the default, never runs agents, and never executes live install/config commands.
• product-harness-setup: explicit opt-in for docs-to-production harness docs and repo-scoped skill templates, including visual product spec pack templates; it never implements app code or enables live providers.

Lane-pack boundaries:

• safe-setup creates lane-pack Markdown docs under docs/harness/** only and must not create .codex/agents.
• Stack-detected lane packs must report activation confidence, matched evidence, and recommendation policy. Weak signals remain advisory candidates instead of default missing-lane work.
• .codex/agents/*.toml custom-agent definitions are generated only by full-orchestration, use harness-* names, and are invoked only when the user explicitly asks Codex to use subagents.
• The lane-pack registry uses universal core lanes plus stack-detected lanes such as UI/UX, backend/API, data, security, performance, infra, AI/ML/CV, docs gardening, and QA.

Skill update behavior

Normal audit runs check this skill's update status by default and write update-status.json plus a Skill update status section in report.md. This check is report-only and never updates skill files. Disable it with:

python3 ~/.codex/skills/harness-engineering-audit/scripts/run_audit.py . --no-check-update

Self-update is explicit and approval-gated. To update this one user-scoped skill installation, run:

python3 ~/.codex/skills/harness-engineering-audit/scripts/run_audit.py . --self-update --update-scope user

The self-update path runs only this command and exits immediately after success:

gh skill install ryne2010/harness-engineering-audit skills/harness-engineering-audit --agent codex --scope user --force

Project-scoped installs should generally be updated intentionally through the repository and reviewed in a PR. Avoid gh skill update --all for this flow because system or manually installed skills may lack GitHub metadata and because this skill should update only itself.

Score dimensions

The report scores:

1. Agent Legibility
2. Instruction Hygiene
3. Docs Authority
4. Vocabulary / Domain Language Control
5. Doc Gardening / Knowledge Base Readiness
6. Validation Truth
7. Harness Feedback Loops
8. Codex Config Readiness
9. Skills Readiness
10. MCP Readiness
11. Hooks / Rules Safety
12. Subagent / OMX Workflow
13. Cross-Agent Compatibility
14. Entropy / Scaffolding Control
15. Production Readiness
16. Symphony Orchestration Readiness
17. Docs-to-Production App Readiness

Scores are strict and intentionally opinionated. A score is not a final truth; it is an evidence-backed signal to guide planning.

The scorecard also includes a lifecycle object and a readiness registry:

• score_schema_version: "2"
• lifecycle.classification
• readiness_registry.categories

The readiness registry exposes additional harness/Symphony categories without silently adding every category as a top-level score dimension.

Fix classification

Classify every recommendation by confidence and risk.

Low-risk / high-confidence

Auto-approved by default for an OMX follow-up pass. AGENTS.md items are P0 and should be handled first:

• trim duplicated root AGENTS.md content
• add or refresh concise root AGENTS.md with docs and validation pointers
• preserve or clarify nested AGENTS.md scope boundaries
• add missing docs index links
• document real validation commands
• add a command registry
• add a validation matrix
• add or link concise glossary/domain-language guidance
• add a doc gardening workflow for indexes, logs, source boundaries, and docs health checks
• clarify deterministic automated checks versus judgment-based automated/human review
• add a generated artifact policy
• add a DX preflight script
• classify generated artifacts
• add a repo operating model doc
• document existing MCP/skills/hooks without changing behavior

Medium-risk

Requires explicit approval before execution:

• restructure docs folders
• add or change MCP servers
• change hook behavior
• change CI validation behavior
• move generated artifacts
• add new skills
• create new repo-level automation

High-risk

Do not execute without explicit approval:

• delete scaffolding or legacy paths
• rewrite CI or deployment workflows
• change package scripts used by production
• remove hooks/rules/MCP servers used by teams
• alter subagent/team workflow semantics
• change security or sandbox settings

Low-typing OMX continuation

Do not make the user copy or retype the long suggested prompts.

After the audit script runs:

1. Read report.md enough to present the summary required below.
2. Use next-step.json / next-step.md to identify the default next stage.
3. In an interactive OMX runtime, present a structured selection:
   • the generated default next stage from next-step.json (default/recommended)
   • Execute auto-approved fixes (only if an execution-ready plan already exists)
   • Review risky questions (only for medium/high-risk ambiguity)
   • Stop after audit
4. If the user selects a stage, continue in the same session with the corresponding generated prompt file in prompts/; do not ask them to paste the command.
5. If no structured selection UI is available, print only the short resume command: $harness-engineering-audit continue.

When the user runs $harness-engineering-audit continue, inspect .codex/reports/harness-engineering-audit/next-step.json. If it exists, resume from its default/recommended stage or present the same selection. If it does not exist, run the audit first.

AGENTS.md priority lane

Treat AGENTS.md improvements as first-class harness fixes:

• Root AGENTS.md is P0 when missing, larger than the concise hot-path target, missing validation commands, missing docs/source-of-truth pointers, or missing nested instruction scope guidance.
• Low-risk AGENTS.md changes are auto-approved. Do not wait for another approval to add, trim, or refresh root AGENTS.md as a concise operating map.
• Keep root AGENTS.md map-like. Move detailed process prose into docs and link to it.
• Preserve nested AGENTS semantics. Do not delete nested instruction files without explicit approval.
• Do not change hooks, MCP, CI, package scripts, security/sandbox settings, source structure, or delete scaffolding as part of the auto-approved AGENTS.md lane.

OMX handoff workflow

After generating omx-handoff.md, the generated prompt files encode this flow:

$ralplan "Read .codex/reports/harness-engineering-audit/prompts/ralplan.md and follow it."

$team "Read .codex/reports/harness-engineering-audit/prompts/team.md and follow it."

$ralph "Read .codex/reports/harness-engineering-audit/prompts/ralph.md and follow it."

$deep-interview "Read .codex/reports/harness-engineering-audit/prompts/deep-interview.md and follow it." # only for medium/high-risk questions

Output discipline

When presenting results, include:

• overall verdict
• score summary
• strongest surfaces
• weakest surfaces
• auto-approved low-risk fixes, especially AGENTS.md P0 items
• medium/high-risk deferred recommendations
• report path
• next OMX selection/resume command

Do not claim the repo is production-ready unless validation truth, instruction hygiene, docs authority, and feedback loops are all strong and evidenced.