# AI Audit > Run a full AI Playbook audit against an existing agentic application or AI feature. Orchestrates Decision Record reverse-engineering, security review, eval audit, launch check, and consolidated remediation planning in one flow. Use when the user explicitly runs /ai-audit or says "audit my AI app", "verify my agent against the playbook", "is my existing AI code production-ready", "review my agentic system", or similar phrasing for verifying existing code (not building new). - Skill: `kumaran-is/ai-audit` (Agent Skill) - Install (CLI): `npx skillmds@latest add kumaran-is/ai-audit` - Raw SKILL.md: https://api.skillmd.com/api/skills/kumaran-is/ai-audit/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: kumaran-is (https://skillmd.com/u/kumaran-is) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/kumaran-is/ai-audit --- # AI Application Audit > **Iron Law:** Do not auto-fix. This skill produces reports and a remediation plan — not code changes. Every finding becomes a separate task for a follow-up session. Run a comprehensive audit of an existing agentic application or AI feature against the AI Playbook. This is for **verifying existing code**, not designing new features. The audit runs in five phases with explicit confirmation gates between each. Each phase produces a written artifact in the repo. The final output is a prioritized remediation plan. ## When to use this vs. the individual skills - Use `/ai-audit` when you want a **full pass** against existing code and you're willing to let it run end-to-end. - Use the individual skills (`/ai-decision-record`, `/ai-launch-check`, `/ai-incident-response`) and subagents (`@ai-security-reviewer`, `@ai-eval-designer`) when you want **finer control** between phases or you're only doing part of the work. ## Operating procedure Walk through the five phases below in order. **Do not skip phases.** Each phase has an explicit confirmation gate where the user can pause, redirect, or abort. Before starting, load the `ai-playbook` skill if it isn't already in context. --- ### Phase 0: Scoping (no artifact) Before any audit work, get clarity on what's being audited. Ask the user: 1. **What is the feature slug?** (Used for directory naming. If unsure, propose one from the repo structure.) 2. **Is this a single feature audit or a multi-feature codebase?** (If multi-feature, ask which feature to audit first — running the full audit on multiple features in one shot is too token-heavy.) 3. **Where is the AI code?** (Path or paths. If unknown, offer to scan and report back before continuing.) 4. **Is there an existing Decision Record?** (Check `docs/ai-decisions/.md`. If yes, this skips Phase 1 and uses it as input.) After confirming scope, summarize the plan: ``` Audit plan for: - AI code location: - Phases to run: 1. Decision Record reverse-engineering (or skip if exists) 2. Security review via @ai-security-reviewer subagent 3. Eval audit via @ai-eval-designer subagent 4. Launch check (gate) 5. Consolidated remediation plan - Estimated wall time: 60–120 minutes depending on codebase size - Estimated token cost: significant (two subagents + main session). Confirm before proceeding. Proceed? (yes / scope-down / abort) ``` **Do not proceed without explicit confirmation.** This audit consumes meaningful tokens and produces artifacts in the repo. --- ### Phase 1: Decision Record (reverse-engineered) **Skip this phase if a Decision Record already exists at `docs/ai-decisions/.md`.** In that case, read it, summarize what it claims, and ask the user to confirm it still reflects reality before moving to Phase 2. If no Decision Record exists, reverse-engineer one: 1. **Scan the AI code.** Identify every AI surface: LLM calls, agent loops, tool definitions, prompts (system / user / templates), RAG/retrieval, model output handlers, side-effect points (writes, sends, payments). 2. **Determine the pattern** (assistant / workflow / agent / autonomous) based on what the code *does*, not what it's named. Multi-step planning with tools is an agent. A single LLM call wrapped in retries is a workflow. 3. **Propose a risk tier** based on actual blast radius. Sum across a session, not per action. Be honest. If the code can send emails to customers, that's at least Tier 3a regardless of how it's framed. 4. **Ask clarifying questions** for what the code can't tell you. Ask one at a time: - Who is the named human owner? - What is the user problem this solves? - What is the cost ceiling per task? - What is the current eval set (if any)? - What is the SLA for human review (if any review queue exists)? - What is the 12-month AI tax estimate? - Where is the kill switch? - When was it last tested? 5. **Write the Decision Record** to `docs/ai-decisions/.md` using the template from the `ai-decision-record` skill / Layer 3 §3.3 of the playbook. 6. **Confirmation gate**: Summarize the Decision Record. Highlight any fields where you had to guess. Ask the user to verify before proceeding. ``` Decision Record drafted at docs/ai-decisions/.md. Honest summary: - Pattern: - Risk tier: - Top three risks I identified: ... - Fields I had to guess (please verify): ... Proceed to Phase 2 (security review)? (yes / edit-record / abort) ``` --- ### Phase 2: Security review **Confirmation before invoking the subagent.** Subagents consume real tokens. Confirm scope: ``` About to invoke @ai-security-reviewer on: - The subagent will run in its own context. Expected output: structured findings report (Critical / High / Medium). Proceed? (yes / narrow-scope / abort) ``` After confirmation, invoke: ``` @ai-security-reviewer audit the AI code at against the ai-playbook security defenses (Layer 2 §2.5) and the adversarial test catalog (Layer 3 §3.2). Focus on: - LLM call sites - Tool definitions and agent loops - RAG/retrieval code - Anywhere user input, retrieved documents, or tool outputs flow into prompts - Anywhere model output triggers side effects Output the standard findings report. For each finding, cite file:line and explain why a prompt-level mitigation is insufficient. Decision Record context: docs/ai-decisions/.md ``` Write the subagent's report to `docs/ai-audits//02-security-review.md`. **Confirmation gate**: ``` Security review complete. Summary: - Critical findings: - High findings: - Medium findings: Top three blockers: ... Full report: docs/ai-audits//02-security-review.md Proceed to Phase 3 (eval audit)? (yes / pause-to-remediate / abort) ``` If there are Critical findings, **recommend pausing to remediate before continuing**. The launch check (Phase 4) will fail on Criticals anyway; fixing them first makes the rest of the audit more useful. --- ### Phase 3: Eval audit **Confirmation before invoking the subagent:** ``` About to invoke @ai-eval-designer on: - Existing tests/evals at: - Risk tier from Decision Record: The subagent will identify eval gaps and generate a starter eval set for the biggest ones. Proceed? (yes / narrow-scope / abort) ``` After confirmation, invoke: ``` @ai-eval-designer audit eval coverage for this agentic app. 1. Find existing tests and evals at: 2. Read the Decision Record at docs/ai-decisions/.md (risk tier: ). 3. Compare current eval coverage against the playbook's risk-tiered minimums (Layer 2 §2.14). 4. Identify gaps across happy path, edge cases, adversarial, negative. 5. Generate a starter eval set at evals//cases/starter.json filling the most critical gaps — especially adversarial cases from Layer 3 §3.2. 6. Output a gap analysis report. ``` Write the report to `docs/ai-audits//03-eval-audit.md`. **Confirmation gate**: ``` Eval audit complete. Summary: - Current eval set size: - Required minimum for Tier : - Gap: cases - Categories most under-covered: ... Starter eval set generated at: evals//cases/starter.json Full report: docs/ai-audits//03-eval-audit.md Proceed to Phase 4 (launch check)? (yes / abort) ``` --- ### Phase 4: Launch check This is the audit's accountability moment. The launch check demands evidence for every item — and on existing code, **expect to fail it on the first pass.** That's the audit doing its job. Run the launch check inline (do not invoke a subagent — this needs to stay in the main session to read Phase 1–3 artifacts): Apply the unified checklist from Layer 4 §4.1 of the playbook. For each item: - Try to verify from the codebase directly first. - Cross-reference Phase 2 (security) and Phase 3 (eval) findings. - Ask the user for evidence only when code inspection is inconclusive. - Mark ✅ / ⚠️ / ❌. **Be strict about evidence.** "We have logging" without pointing to the logging schema doesn't count. "We have a kill switch" without pointing to a tested feature flag doesn't count. Write the launch check report to `docs/ai-audits//04-launch-check.md`. Include: - Decision: GO / NO-GO / GO-WITH-EXCEPTIONS - Every checklist item with status and evidence - Cross-references to Phase 2 and Phase 3 findings **Confirmation gate**: ``` Launch check complete. Decision: - ✅ Verified: - ⚠️ Partial: - ❌ Missing: Top blockers: ... Full report: docs/ai-audits//04-launch-check.md Proceed to Phase 5 (consolidated remediation plan)? (yes / abort) ``` --- ### Phase 5: Consolidated remediation plan Synthesize findings from all four prior phases into one prioritized backlog. Read the artifacts: - `docs/ai-decisions/.md` (Decision Record) - `docs/ai-audits//02-security-review.md` - `docs/ai-audits//03-eval-audit.md` - `docs/ai-audits//04-launch-check.md` Group every finding into one of four priority tiers: 1. **Critical — fix before any further production traffic.** Security criticals; missing or untested kill switch; missing output contract on Tier-3+ features; missing tenant isolation; secrets in model context. 2. **High — fix within 2 weeks.** High security findings; eval set below risk-tiered floor; missing observability; missing incident response plan; rollback path not tested. 3. **Medium — schedule in the next quarter.** Medium security findings; drift detection gaps; vendor abstraction; multi-modal handling refinements; AI tax not estimated. 4. **Structural — design debt to track.** Full reference architecture gaps; eval flywheel not in place; canary deployment infrastructure; full Layer 2 coverage. For each item, include: - **Source**: which audit phase surfaced it - **Playbook reference**: which section/layer - **Concrete next action**: what to actually do - **Suggested owner role** (named human in Decision Record; if unspecified, propose the role) - **Acceptance criteria**: how you'll know it's done Write to `docs/ai-audits//05-remediation-plan.md`. **Final output to user**: ``` AI Audit Complete Feature: Pattern: Risk tier: Launch check decision: Findings: - Critical: items - High: items - Medium: items - Structural: items Top three things to fix this week: 1. ... 2. ... 3. ... Full artifacts: - Decision Record: docs/ai-decisions/.md - Security review: docs/ai-audits//02-security-review.md - Eval audit: docs/ai-audits//03-eval-audit.md - Launch check: docs/ai-audits//04-launch-check.md - Remediation plan: docs/ai-audits//05-remediation-plan.md Suggested next step: - If Critical items: pause, fix them, re-run `/ai-audit` to verify - If High items: schedule with owners, re-run `/ai-audit` in 2 weeks - If only Medium/Structural: ship with exceptions filed in Decision Record; track in regular planning ``` --- ## Default stance - **Be strict about evidence in Phase 4.** The whole audit's value depends on the launch check being honest. If you accept verbal assertions, the audit produces false confidence. - **Treat the first run as a baseline, not a verdict.** Existing code rarely passes a fresh audit. The output is a remediation backlog, not a launch decision. - **Default to recommending pause-to-remediate after Phase 2 if there are Criticals.** Continuing the audit past unfixed Critical security findings makes the rest of the work less useful. - **Confirmation gates are not optional.** Even if the user said "yes go through everything" at the start, each gate is a chance to course-correct. Honor them. - **Don't auto-fix.** This skill produces reports and plans, not code changes. Remediation is a separate conversation per finding. ## What this skill does NOT do - It does not implement fixes. Each finding becomes a task; the human (or a follow-up Claude Code session) does the work. - It does not replace ongoing eval maintenance, security review, or incident response. It's a snapshot audit; the playbook's lifecycle controls (eval flywheel, drift detection, incident learning) are continuous. - It does not certify production readiness. The launch check decision is advisory; final ship/no-ship is a human + business call informed by this report. - It does not run on multiple features in parallel. Audit one feature at a time. Repeat for the next. ## Verify (after Phase 5 is complete) ``` ✅ docs/ai-decisions/.md exists (created or confirmed in Phase 1) ✅ docs/ai-audits//02-security-review.md written ✅ docs/ai-audits//03-eval-audit.md written ✅ docs/ai-audits//04-launch-check.md written with GO/NO-GO decision ✅ docs/ai-audits//05-remediation-plan.md written with priority tiers ✅ No audit phase was silently skipped — every abort was explicit and user-confirmed ``` ## If this skill cannot proceed ``` Abort condition: user declines scope confirmation at Phase 0 → Stop. Do not run any phases. → Output: "Audit cancelled at scoping. No artifacts written." Abort condition: Critical security findings in Phase 2 → Recommend: "Pause audit. Fix Critical findings first. Re-run /ai-audit to continue." → Do not mark audit complete — it is paused, not done. Abort condition: Decision Record missing and user refuses to reverse-engineer one → Stop at Phase 1. Output: "Cannot proceed without a Decision Record. Run /ai-decision-record first." ``` ## References - AI Playbook: `.claude/skills/ai-playbook/playbook.md` (Layers 1–4) - Decision Record template: `.claude/skills/ai-decision-record/SKILL.md` - Launch checklist: `.claude/skills/ai-launch-check/SKILL.md` (Layer 4 §4.1) - Security adversarial catalog: Layer 3 §3.2 of playbook.md - Eval sizing table: Layer 2 §2.14 of playbook.md - OWASP LLM Top 10: https://owasp.org/www-project-top-10-for-large-language-model-applications/