AI ROI Audit Skill
Every org now spends real money on AI tools, and most justify it with adoption counts ("80% weekly active!") — which measure enthusiasm, not return. This skill audits what the spend returned, using methods that survive a sceptical CFO: baselines, counterfactuals, and quality deltas, with "we can't know yet" said out loud where it's true.
What This Skill Produces
- A per-tool verdict table: keep / consolidate / renegotiate / cut, each with its evidence
- The measurement behind each number — method, baseline, confidence — so the audit is checkable
- A hidden-cost ledger (the part vendor ROI decks omit)
- A baseline plan for every "unknown", so next year's audit has data
Required Inputs
Ask for (if not already provided):
- The AI tool inventory with costs: subscriptions, API spend, seats — and utilisation if known
- What each tool was bought to do (the promised outcome, from the original business case if it exists)
- Available evidence: usage data, before/after metrics, time studies, quality data, anecdotes (labelled as anecdotes)
- The decision at stake: renewal? consolidation? budget defence? (calibrates depth)
Audit Method
- Reconstruct the promise. Per tool: what outcome justified the purchase — time saved, quality improved, headcount avoided, revenue created? A tool without a stated outcome gets audited against the best-fit guess, flagged as retrofitted.
- Score with the strongest method the evidence allows, in descending order of credibility:
- Natural experiment — teams/periods with vs without the tool, same work (best available in most orgs)
- Before/after with baseline — the metric before adoption vs after, seasonality noted
- Task-level time study — 10-20 real tasks timed with/without (cheap to run during the audit — do it rather than skip to tier 4)
- Structured self-report — users estimating time saved, discounted (self-reported AI savings run ~2× actuals; say so)
Never present a tier-4 number with tier-1 confidence. Every figure carries its method and a confidence label.
- Count the hidden costs. Verification time (humans checking AI output), rework from AI errors that shipped, licence sprawl (seats bought > seats active), integration/prompt-maintenance time, and training time. These come off the gross benefit — an ROI audit that skips them is a vendor deck.
- Convert honestly. Time saved → money only via a stated loaded rate and a stated assumption about what the time became (more output? earlier finishes? — different values). "Saved 400 hours" that nobody redeployed is capacity, not cash; label which one you're claiming.
- Verdict per tool. Keep (positive with tier ≤2 evidence) · Consolidate (positive but duplicative — name the overlap) · Renegotiate (positive but mispriced vs utilisation) · Cut (negative or unmeasurable after a fair baseline attempt). Ties break toward the tool with a measurement plan.
- Leave the audit better than you found it. Every "unknown" verdict gets a baseline plan: the metric, how it's instrumented, and the review date. The first audit is mostly this; that's a finding, not a failure.
Output Format
AI ROI Audit: [org/team] — [period]
Total AI spend: [sum] · Verdict summary: [n keep / n consolidate / n renegotiate / n cut / n unknown]
| Tool |
Annual cost |
Promised outcome |
Measured return |
Method (tier) |
Confidence |
Verdict |
Hidden-cost ledger: [verification, rework, sprawl, maintenance — quantified where possible, listed where not]
The math shown: [for each material number: baseline, method, conversion assumptions]
Baseline plan for the unknowns: [tool → metric → instrumentation → review date]
One-paragraph CFO summary: [net position, the two decisions to make, and what will be measurable by next audit]
Quality Checks
Anti-Patterns
Example Trigger Phrases
- "What did our AI tools actually return?"
- "Should we renew these AI contracts?"
- "We pay for three overlapping AI subscriptions: which do we keep?"
- "Build the measurement plan before the next AI spend."
1---2name: ai-roi-audit3description: "Audit whether the organisation's AI spend actually paid — measured against baselines, not vendor math or vibes. Use when a CFO asks what the AI tools returned, when renewing AI contracts, when consolidating overlapping AI subscriptions, or to build the measurement plan before the next spend. Produces an ROI audit with per-tool verdicts (keep/consolidate/cut), the honest-measurement method behind each number, and a baseline plan for whatever can't be scored yet. To forecast ROI before an investment use roi-estimator; this skill measures what already happened."4disable-model-invocation: true # skill-optimizer5metadata:6 source: https://skillmd.com/skills/slipcast-dev/ai-roi-audit7---89# AI ROI Audit Skill1011Every org now spends real money on AI tools, and most justify it with adoption counts ("80% weekly active!") — which measure enthusiasm, not return. This skill audits what the spend *returned*, using methods that survive a sceptical CFO: baselines, counterfactuals, and quality deltas, with "we can't know yet" said out loud where it's true.1213## What This Skill Produces1415- A **per-tool verdict table**: keep / consolidate / renegotiate / cut, each with its evidence16- The **measurement behind each number** — method, baseline, confidence — so the audit is checkable17- A **hidden-cost ledger** (the part vendor ROI decks omit)18- A **baseline plan** for every "unknown", so next year's audit has data1920## Required Inputs2122Ask for (if not already provided):23- **The AI tool inventory with costs**: subscriptions, API spend, seats — and utilisation if known24- **What each tool was bought to do** (the promised outcome, from the original business case if it exists)25- **Available evidence**: usage data, before/after metrics, time studies, quality data, anecdotes (labelled as anecdotes)26- **The decision at stake**: renewal? consolidation? budget defence? (calibrates depth)2728## Audit Method29301. **Reconstruct the promise.** Per tool: what outcome justified the purchase — time saved, quality improved, headcount avoided, revenue created? A tool without a stated outcome gets audited against the best-fit guess, *flagged as retrofitted*.312. **Score with the strongest method the evidence allows**, in descending order of credibility:32 - **Natural experiment** — teams/periods with vs without the tool, same work (best available in most orgs)33 - **Before/after with baseline** — the metric before adoption vs after, seasonality noted34 - **Task-level time study** — 10-20 real tasks timed with/without (cheap to run *during* the audit — do it rather than skip to tier 4)35 - **Structured self-report** — users estimating time saved, discounted (self-reported AI savings run ~2× actuals; say so)36 Never present a tier-4 number with tier-1 confidence. Every figure carries its method and a confidence label.373. **Count the hidden costs.** Verification time (humans checking AI output), rework from AI errors that shipped, licence sprawl (seats bought > seats active), integration/prompt-maintenance time, and training time. These come off the gross benefit — an ROI audit that skips them is a vendor deck.384. **Convert honestly.** Time saved → money only via a stated loaded rate *and* a stated assumption about what the time became (more output? earlier finishes? — different values). "Saved 400 hours" that nobody redeployed is capacity, not cash; label which one you're claiming.395. **Verdict per tool.** Keep (positive with tier ≤2 evidence) · Consolidate (positive but duplicative — name the overlap) · Renegotiate (positive but mispriced vs utilisation) · Cut (negative or unmeasurable after a fair baseline attempt). Ties break toward the tool with a measurement plan.406. **Leave the audit better than you found it.** Every "unknown" verdict gets a baseline plan: the metric, how it's instrumented, and the review date. The first audit is mostly this; that's a finding, not a failure.4142## Output Format4344### AI ROI Audit: [org/team] — [period]4546**Total AI spend:** [sum] · **Verdict summary:** [n keep / n consolidate / n renegotiate / n cut / n unknown]4748| Tool | Annual cost | Promised outcome | Measured return | Method (tier) | Confidence | Verdict |49|---|---|---|---|---|---|---|5051**Hidden-cost ledger:** [verification, rework, sprawl, maintenance — quantified where possible, listed where not]5253**The math shown:** [for each material number: baseline, method, conversion assumptions]5455**Baseline plan for the unknowns:** [tool → metric → instrumentation → review date]5657**One-paragraph CFO summary:** [net position, the two decisions to make, and what will be measurable by next audit]5859## Quality Checks6061- [ ] Every figure carries its measurement method and confidence — no naked numbers62- [ ] Self-reported savings are discounted and labelled as self-reported63- [ ] Hidden costs appear as line items, not a caveat sentence64- [ ] Time→money conversions state the loaded rate and the capacity-vs-cash claim65- [ ] Every "unknown" has a baseline plan with a date — the audit compounds6667## Anti-Patterns6869- [ ] Do not use adoption or engagement as return — usage is a cost signal until an outcome moves70- [ ] Do not accept vendor ROI calculators as evidence — reconstruct from your own data or score it unknown71- [ ] Do not average across tools into one triumphant number — the verdict is per-tool or it decides nothing72- [ ] Do not claim headcount avoidance without the counterfactual hiring plan that was actually cancelled73- [ ] Do not punish honest "unknowns" by cutting them reflexively — cut requires a *failed* measurement attempt, not a missing one7475## Example Trigger Phrases7677- "What did our AI tools actually return?"78- "Should we renew these AI contracts?"79- "We pay for three overlapping AI subscriptions: which do we keep?"80- "Build the measurement plan before the next AI spend."