# Audit AI Agents > Research, scope, design, execute, and report rigorous audits of AI agent systems. Use for Agent audit literature reviews, frontier-lab comparisons, alignment or hidden-objective audits, agent security red teams, AI-control and monitor evaluations, tool/MCP/permission audits, production observability and incident forensics, benchmark audits, governance assurance, audit plans, evidence registers, reproducibility reviews, or research-gap generation. Distinguish model evaluation from whole-system auditing and pre-deployment propensity testing from runtime control evidence. - Skill: `pnx2003/audit-ai-agents` (Agent Skill, multi-file: 13 files) - Install (CLI): `npx skillmds@latest add pnx2003/audit-ai-agents` - Raw SKILL.md: https://api.skillmd.com/api/skills/pnx2003/audit-ai-agents/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: pnx2003 (https://skillmd.com/u/pnx2003) - Updated: 2026-09-22 - Page: https://skillmd.com/skills/pnx2003/audit-ai-agents --- # Audit AI Agents ## Overview Treat an agent audit as an evidence-producing investigation of a socio-technical system, not as a single benchmark score. Map the audited boundary, threat model, claims, controls, traces, tests, and residual uncertainty across the model, harness, tools, memory, identities, environment, humans, and operational process. This skill combines literature research, code inspection, audit design, evidence grading, incident-aware analysis, and research-gap generation. It favors primary artifacts and reproducible tests, records negative and conflicting evidence, and states what the available access cannot establish. ## Route the request 1. For a literature review or frontier-company map, read `references/taxonomy.md`, `references/frontier-labs.md`, `references/literature.md`, and `references/update-protocol.md`. 2. For an audit plan or execution, read `references/taxonomy.md` and `references/audit-playbook.md`; add `references/codebases.md` when selecting tools or benchmarks. 3. For production monitoring, evidence packages, or an incident, read `references/audit-playbook.md` and the provenance/forensics sections of `references/literature.md`. 4. For a paper, benchmark, or repository review, read `references/codebases.md` and the evidence-extraction protocol in `references/update-protocol.md`. 5. For new research ideas, first complete the evidence map, then read `references/research-gaps.md`. Never generate novelty claims from memory alone. 6. Query the curated corpus with `python3 scripts/query_corpus.py --help`. Treat it as a dated seed set, not a substitute for a current search. ## Establish the audit contract Before collecting evidence, write a compact contract containing: - **Decision:** the concrete release, deployment, procurement, incident, research, or assurance decision the audit must support. - **System boundary:** exact model/version, system prompt, harness, tools and protocols, identities and permissions, memory/RAG, environment, external services, human approvals, monitors, and logging stack. - **Stage:** design review, pre-deployment, runtime/continuous, post-incident, or third-party assurance. - **Access:** black-box; gray-box with prompts, traces, logs, training data, or internals; or white-box with weights/activations. List unavailable evidence. - **Threat model:** actor, objective, capability, knowledge, access, persistence, collusion, and acceptable impact threshold. - **Claims:** separate capability, propensity, control effectiveness, security, compliance, and incident claims. - **Freshness date:** search cutoff, model build, harness commit, policy version, tool versions, and audit environment. If the boundary or decision is missing, ask only for information that would materially change the test plan. Otherwise state a narrow assumption and proceed. ## Run the evidence workflow ### 1. Build an audit register Create rows for each relevant combination of system layer, risk, lifecycle stage, expected control, evidence source, test, metric, owner, status, and residual uncertainty. Use the dimensions and risk families in `references/taxonomy.md`. Do not collapse these activities: - capability evaluation: what the agent can do; - propensity or alignment audit: what it is inclined to do; - AI-control evaluation: whether harm remains containable even if the agent is untrusted; - security red team: whether an adversary can manipulate or compromise the system; - runtime monitoring: what the deployed system did or is doing; - assurance audit: whether control design and operation meet stated obligations; - incident forensics: what happened, why, and with what evidence integrity. ### 2. Search systematically Follow `references/update-protocol.md`. Start with official research pages, papers, system cards, standards, incident reports, repositories, datasets, and evaluation code. Then use surveys and citation graphs for coverage. Record exact queries, repositories, dates, inclusion decisions, and unresolved names. Never promise literally “all articles.” Report a search-complete set relative to named databases, organizations, query families, languages, artifact types, and cutoff date. ### 3. Extract comparable evidence For every included artifact capture: - exact claim and audit object; - authors/organization, date, artifact status, and funding or first-party relationship; - model, harness, environment, access level, threat model, and adversary; - task construction, sample size, seeds/runs, baselines, controls, scorer, and human review; - capability ceiling, attack success, monitor discrimination, safety and utility, severity, uncertainty, and audit cost where reported; - realism, evaluation-awareness, contamination, independence, transfer, and reproduction limits; - code/data availability, license, infrastructure, credentials, and last maintenance signal. Label findings `confirmed`, `supported`, `indicative`, `company-reported`, `disputed`, or `unknown`. A company blog can be primary evidence of what that company reports, but not independent validation of the underlying safety claim. ### 4. Inspect implementation, not just prose When code is available, inspect entry points, dependency locks, environment isolation, privilege boundaries, tool mocks, scorer implementation, hidden-test protection, transcript format, resume behavior, randomization, seed handling, telemetry, and expected compute/API cost. Run a safe smoke test when feasible. Record deviations between the paper, current branch, released data, and claimed setup. Prefer synthetic credentials, sandboxed services, read-only replicas, and reversible actions. Do not test exploits against live systems, third parties, real user data, or production credentials without explicit authorization and an agreed rules-of-engagement document. ### 5. Design discriminating tests Use at least one benign baseline, one capability/control condition, one adversarial condition, and one ablation that distinguishes the favored explanation from a plausible alternative. Add multiple trials for stochastic agents. Measure both safety and task utility. For monitor audits, estimate false negatives and false positives under realistic base rates; include monitor-independent ground truth when possible; vary information channels; test evaluator awareness, strategic obfuscation, correlated failure, and monitor gaming. For autonomous actions, capture intended plan, observations, tool calls, arguments, results, state changes, approvals, identities, timestamps, policy decisions, and final outcomes. ### 6. Triangulate and report uncertainty Require at least two independent evidence modes for high-impact conclusions when feasible: behavioral tests, trace review, human investigation, system logs, static analysis, activation/weight tools, training-data analysis, or incident evidence. Explain conflicts instead of averaging them away. Use calibrated language. A passed test supports only the tested configuration and threat model. A failed attack is not evidence of absence if capability, access, realism, or search budget was inadequate. ## Output contract Produce the smallest useful set of these sections: 1. **Executive decision:** decision, tested boundary, major finding, and highest residual risk. 2. **Scope and exclusions:** versions, access, threat model, cutoff date, and explicit non-claims. 3. **Evidence map:** comparable source table with evidence grade and independence. 4. **System and control map:** layers, authority, trust boundaries, and observable evidence. 5. **Test plan or results:** hypotheses, procedures, baselines, trials, scorers, safety/utility metrics, findings, and artifacts. 6. **Control gaps:** affected layer, exploit or failure path, severity, likelihood, detectability, evidence strength, and owner. 7. **Residual uncertainty:** unavailable data, realism and transfer gaps, monitor blind spots, and untested changes. 8. **Actions:** release gate, containment, monitoring, remediation, re-test trigger, and evidence-retention requirement. 9. **Research gaps:** only after nearest-prior-art and synonym re-search, using `references/research-gaps.md`. Keep claims linked to sources or audit artifacts. Separate direct observation, inference, and recommendation. Include dates and version identifiers in every table that could otherwise become stale. ## Minimum quality gate Do not call work an agent audit unless it has: - an explicit system boundary and decision; - a threat model and access statement; - a versioned evidence register; - tests or controls tied to specific claims; - safety and utility measurements, or a reason one is inapplicable; - scorer and ground-truth limitations; - stochastic variance or an explicit single-run limitation; - evidence integrity and retention notes; - residual uncertainty and re-audit triggers; - a distinction between first-party claims and independent evidence. Reject one-number summaries when they conceal severity, scope, base rate, utility cost, or monitor error. Do not infer real-deployment safety from a synthetic model organism, benchmark, or simulated environment without a transfer argument.