alexe-ev Define AI Success Metrics
Purpose
Help teams define a complete set of success metrics for an AI feature — covering model quality, user experience, business impact, and safety — so that success and failure can be measured objectively.
Skill type
Conceptual skill
Use this skill when
- An AI feature is being designed and success criteria haven't been defined
- A launched AI feature lacks a clear way to measure performance
- Stakeholders disagree on how to evaluate the AI feature's performance
- An AI feature's metrics need to be separated into quality, UX, business, and safety dimensions
Do not use this skill when
- General product metrics are needed (use design-product-metrics)
- AI quality monitoring systems are being designed (use evaluate-ai-quality-monitoring)
Required inputs
- AI feature description and target user
- Business goal the AI feature serves
Optional inputs
- Model capability assessment
- Human-in-the-loop design
- Prior user research on the problem area
- Regulatory context
Upstream context
Works best when:
- AI value is framed
- Model choice is made
- HITL design exists
Downstream handoff
Output can feed:
- evaluate-ai-quality-monitoring (success metrics → monitoring system)
- build-decision-dashboard (AI metrics → dashboard)
- design-experiment-plan (metrics → experiment success criteria)
Instructions
- Define model quality metrics: how do you measure if the AI output is good? (accuracy, relevance, hallucination rate, etc.)
- Define user experience metrics: how do users respond to the AI? (adoption rate, trust signals, feedback, task completion).
- Define business impact metrics: how does the AI feature affect product KPIs? (retention, efficiency, revenue, support deflection).
- Define safety metrics: what could go wrong and how would you detect it? (error rates, harmful output flags, user complaint rate).
- Set baseline and target values for each metric where possible.
- Define review cadence and who owns each metric.
Output
Provide:
- Model quality metrics with definitions and measurement approach
- User experience metrics
- Business impact metrics
- Safety and risk metrics
- Baseline and target values (where available)
- Primary metric (the one that best captures overall feature success)
- Review cadence and metric owners
Risks / caveats
- Model quality metrics and user experience metrics are often uncorrelated — measure both
- Safety metrics must be defined before launch, not after an incident
- Don't choose metrics that are easy to measure over metrics that matter