alexe-ev Conduct AI Audit
Purpose
Help teams run a structured audit of a deployed or near-deployed AI system to evaluate compliance with responsible AI standards and identify gaps that need to be addressed.
Skill type
Conceptual skill
Use this skill when
- A deployed AI feature needs to be reviewed against responsible AI standards
- A governance process requires periodic AI audits
- An AI feature has been flagged for bias, fairness, or safety concerns
- Regulatory or legal requirements mandate an AI audit
- An AI feature is being scaled and needs a formal readiness review
Do not use this skill when
- AI governance standards haven't been defined yet (use govern-responsible-ai first)
- The goal is ongoing quality monitoring (use evaluate-ai-quality-monitoring)
Required inputs
- AI feature or system to audit
- Responsible AI standards to audit against (company, regulatory, or industry standards)
Optional inputs
- AI governance framework
- Quality monitoring data
- User feedback or complaints about the AI
- Prior audits or assessments
Upstream context
Works best when:
- AI governance principles are defined
- AI quality monitoring is in place
Downstream handoff
Output can feed:
- govern-responsible-ai (audit findings → governance improvements)
- plan-risk-mitigation (audit gaps → risk mitigations)
- evaluate-ai-quality-monitoring (audit reveals monitoring gaps)
Instructions
- Define the audit scope: which AI feature, which standards, what time period.
- Gather evidence: model documentation, training data provenance, quality metrics, user feedback, incident log.
- Audit against each responsible AI dimension: fairness, transparency, accountability, safety, privacy.
- Identify gaps: where does the system not meet the standard?
- Assess severity of each gap: critical / major / minor.
- Produce findings and recommendations.
- Define remediation timeline and ownership.
Output
Provide:
- Audit scope and standards applied
- Evidence gathered
- Findings by dimension (fairness, transparency, accountability, safety, privacy)
- Gap severity classification
- Remediation recommendations with owners and timelines
- Overall audit verdict: compliant / conditionally compliant / non-compliant
- Next audit schedule
Risks / caveats
- AI audits are snapshots — model behavior can change with data drift; schedule re-audits
- Audits without executive visibility don't drive remediation
- Fairness audits require defining "fair" explicitly — it's not a universal standard