talgacapri Why this exists, and what I'd change
Why it exists. RICE doesn't work for AI features. Reach, impact, confidence, and effort assume predictable outputs. AI features live on a spectrum of "how well does it work" and need a different lens. This skill replaces RICE with four AI-native scores: model confidence, cost per inference, eval-readiness, and fallback quality.
Design tradeoffs.
- Four scores, not one composite. I don't roll them into a single number. Cost: harder to sort a backlog. PMs who want a ranked list have to apply their own weighting.
- Forces a ship/MVP/prototype/kill decision instead of a score. Every feature ends with a verb, not a number. Cost: less satisfying for execs who like a "RICE score of 47" answer.
- Eval-readiness is a hard gate. If you can't measure model quality on production data, the framework blocks shipping. Cost: real, slow features stuck in MVP because golden examples don't exist yet.
What I'd change. Add a calibration loop. After three AI features ship, compare predicted confidence against actual quality, and adjust the framework's thresholds based on the team's track record.
AI Feature Prioritization: Confidence-Gated Framework
When to use: When prioritizing AI-powered features, deciding what to ship vs prototype, or building an AI product roadmap. Traditional frameworks (RICE, WSJF, MoSCoW, ICE) break down for AI because they assume predictable impact and binary ship/no-ship decisions. AI features live on a spectrum of "how well does it work."
Framework source: Ashutosh Singh's AI Feature Prioritization framework (Confidence-Gated Prioritization)
Trigger phrases: "prioritize AI features", "should we ship this AI feature", "AI roadmap priority", "RICE for AI", "how confident is the model", "cost per inference", "eval-readiness", "AI prioritization matrix", "confidence-gated", "how do I prioritize this AI product"
Why Old Frameworks Fail for AI
| Framework | Assumption | Why It Breaks for AI |
|---|---|---|
| RICE | You can estimate impact before launch | AI features have confidence intervals, not fixed impact |
| WSJF | Features are binary: shipped/not shipped | AI features work on a spectrum of "how well does it work" |
| MoSCoW | Scope is clear and fixed | AI scope depends on model capability. Changes weekly. |
| ICE | Confidence is about team certainty | AI confidence is about model accuracy. Different concept. |
The core problem: A recommendation engine might work 92% of the time. Or 67%. You won't know until it's live. Old frameworks treat features as binary: shipped or not shipped. AI features are on a spectrum of "how well does it work."
The Four Lenses
1. Confidence-Gated Prioritization
Before you estimate impact, estimate model confidence.
The question: How confident are you that the model can do this task at production quality?
| Confidence Level | Definition | Action |
|---|---|---|
| High (90%+) | Model reliably handles this task with minimal errors | Prioritize like a normal feature. Ship it. Standard sprint planning. |
| Medium (70-89%) | Model works most of the time but has notable failure modes | Build an MVP, test with real users, measure. MVP to 5% of users. Measure for 2 weeks. Decide. |
| Low (below 70%) | Model struggles or is inconsistent | Prototype only. Do not put on roadmap. |
How to estimate confidence:
- Run the feature against 50-100 representative inputs
- Have domain experts grade the outputs
- Calculate the percentage that meets "production quality" bar
- If you can't run this test, your confidence is Low by default
Decision rule: Never roadmap a Low confidence feature. Prototype it, learn, revisit when model capability improves.
2. Cost-per-Inference Ranking
Every AI feature has a running cost. This is fundamentally different from traditional software where marginal cost is near zero.
The question: What does each API call cost, and does the business case support it?
How to calculate:
- Estimate calls per user per month
- Multiply by cost per call (API pricing, compute, etc.)
- Get cost per user per month for this feature
- Compare to the revenue or value the feature generates
Decision rules:
| Cost/Inference | Required Business Case |
|---|---|
| < $0.01/user/mo | Ship freely. Cost is negligible. |
| $0.01 - $0.10/user/mo | Normal ROI justification. Show 3-5x return. |
| $0.10 - $0.50/user/mo | Strong ROI required. Show 10x return. VP approval needed. |
| > $0.50/user/mo | Requires VP approval + business case showing 10x ROI |
Add "cost per inference" as a column in your prioritization spreadsheet. Most PMs skip this. Their CFO won't.
Pro tip: Costs drop over time as models get cheaper. A feature that's too expensive today might be viable in 6 months. Track it, don't kill it.
3. Eval-Readiness Scoring
If you can't measure whether an AI feature is working, you can't improve it. And you can't justify keeping it.
The question: Can you measure if this feature works?
Eval-readiness checklist:
| Eval Component | Question | Status |
|---|---|---|
| Golden dataset | Do you have labeled test cases to score against? | Yes / No / Partial |
| Automated eval | Can you run evals without human review? | Yes / No / Partial |
| Success metric | Is the metric clearly defined (precision, recall, user satisfaction)? | Yes / No |
| Feedback loop | Can users flag bad outputs so you improve? | Yes / No |
| Baseline | Do you know performance BEFORE the AI feature? | Yes / No |
Scoring:
- 5/5 (Eval-ready): Ship with confidence. You can measure improvement.
- 3-4/5 (Mostly ready): Ship, but invest in filling gaps first.
- 1-2/5 (Not ready): No eval = no improvement. Build evals first, then decide.
- 0/5: Do not ship. You are flying blind.
The hard truth: If you can't eval it, you can't improve it. Features without evals are features you'll regret launching.
4. Fallback Quality Assessment
What happens when the model fails? This is the question that separates mature AI products from reckless ones.
The question: Is the fallback graceful (show nothing) or catastrophic (show wrong info)?
| Fallback Type | Definition | Confidence Threshold |
|---|---|---|
| Graceful | Feature hides or shows generic content when model fails. User doesn't notice. | Standard threshold (70%+ confidence to ship) |
| Informational | Feature shows a result that might be slightly wrong. User can verify. | Higher threshold (80%+ to ship). Add disclaimers. |
| Catastrophic | Wrong output causes real harm: bad financial advice, incorrect medical info, wrong transaction. | Raise confidence threshold to 95% before shipping. Add human-in-the-loop review. |
Decision rule: Features with catastrophic fallbacks need higher confidence thresholds before shipping. Period.
Fallback strategies:
- Graceful: Hide the feature, show a generic alternative, say "I don't have enough info"
- Informational: Show the AI output with a confidence indicator, let user verify
- Catastrophic: Require human review before showing output, or don't launch until 95%+ confidence
The AI Prioritization Matrix
Combine confidence and impact into a 2x2:
| High Impact | Low Impact | |
|---|---|---|
| High Confidence (90%+) | MUST SHIP - Prioritize like a normal feature | PRIORITIZE - Ship when capacity allows |
| Medium Confidence (70-89%) | MVP TEST - Build MVP, test with real users, measure | BACKLOG - Low priority. Revisit quarterly. |
| Low Confidence (below 70%) | PROTOTYPE ONLY - Don't roadmap. Explore and learn. | KILL - Not worth the investment right now. |
New Columns for Your Prioritization Sheet
Add these four columns to your existing prioritization spreadsheet:
| Feature | Confidence | Cost/Inference | Eval Ready | Fallback | Impact | Decision |
|---|---|---|---|---|---|---|
| AI search | 92% | $0.03 | Ready | Graceful | High | Ship |
| AI summaries | 78% | $0.03 | Ready | Informational | High | MVP test |
| AI agent | 55% | $0.30 | Not ready | Catastrophic | High | Prototype only |
| AI pricing | 45% | $0.08 | Partial | Catastrophic | Medium | Kill |
How to Run an AI Prioritization Session
Step 1: List All AI Features
Gather every AI feature on your roadmap or backlog. Include proposed, in-progress, and shipped features.
Step 2: Score Each Feature on Four Lenses
For each feature, answer:
- Confidence: What % of the time does the model get this right at production quality? (Run eval if possible, estimate if not)
- Cost/Inference: What does it cost per user per month? (API costs + compute + any human review)
- Eval-Readiness: Score 0-5 on the eval checklist
- Fallback Type: Graceful, Informational, or Catastrophic?
Step 3: Apply Decision Rules
| If... | Then... |
|---|---|
| Confidence > 90% AND eval ready | Ship it. Standard sprint planning. |
| Confidence 70-89% | MVP to 5% of users. Measure for 2 weeks. Decide. |
| Confidence < 70% | Prototype only. Do not put on roadmap. |
| No eval dataset exists | Stop. Build evals first. Then decide. |
| Cost/inference > $0.10 | Requires VP approval + business case showing 10x ROI |
| Catastrophic fallback | Raise confidence threshold to 95% before shipping |
Step 4: Build the Prioritized Roadmap
- Immediate ship: High confidence + eval ready + acceptable cost
- MVP test: Medium confidence + eval ready (or close)
- Eval investment: Features worth pursuing but eval not ready (build evals first)
- Prototype backlog: Low confidence features to explore when capacity allows
- Kill list: Low confidence + low impact + high cost. Remove from roadmap.
Applying This in High-Stakes Domains
When prioritizing AI features in regulated or high-stakes domains (finance, healthcare, legal, education), these lenses are especially critical:
Fallback risk is elevated. High-stakes advice means wrong outputs are catastrophic, not just annoying. Apply the 95% confidence threshold and define every fallback before ship.
Cost per inference matters. Frequent interaction means even $0.05/call adds up fast across a user base. Model cost per feature per user per month.
Eval-readiness is non-negotiable. Domain accuracy requires golden datasets. Before shipping any AI skill, build test cases with known-correct outputs.
Confidence varies by skill type:
- Structured data + clear rules — likely high confidence
- Subjective or market-dependent advice — likely medium confidence
- Open-ended, emotional, or behavioral output — likely low-medium confidence
- Rule-based with edge cases — needs careful evaluation
Quick Start
When the PM says "prioritize this AI feature" or "should we ship this AI feature," run through this sequence:
- Identify all AI features to prioritize (or focus on the one in question)
- Score each on 4 lenses (Confidence, Cost, Eval-Readiness, Fallback)
- Apply the prioritization matrix (Must Ship / MVP Test / Prototype Only / Kill)
- Apply decision rules for edge cases
- Output a prioritized table with clear ship/test/prototype/kill recommendations
- Flag any features with catastrophic fallbacks that need elevated thresholds
Output Format
When delivering AI prioritization output, include:
AI Feature Prioritization Summary
Date: [today] Features assessed: [count]
Prioritization Table
| # | Feature | Confidence | Cost/User/Mo | Eval Score | Fallback | Impact | Decision |
|---|---|---|---|---|---|---|---|
| 1 | [name] | [%] | [$] | [0-5] | [type] | [H/M/L] | [Ship/MVP/Proto/Kill] |
Ship Now (High Confidence + Eval Ready)
- [features and why]
MVP Test (Medium Confidence)
- [features, test plan, success criteria]
Build Evals First
- [features that need eval investment before deciding]
Prototype Only (Low Confidence)
- [features to explore, not roadmap]
Kill (Not Worth Pursuing)
- [features and why]
Cost Alerts
- [any features where inference cost needs VP approval]
Catastrophic Fallback Warnings
- [any features with dangerous failure modes]
Related Skills
/prioritize- LNO Framework for PM task prioritization (different: tasks vs features)/impact-sizing- Quantify feature value with driver trees (use alongside this for the "Impact" column)/experiment-decision- When to A/B test vs ship (complements the MVP Test decision)/feature-metrics- Define success metrics (feeds into Eval-Readiness scoring)/mece- Structure AI feature categories exhaustively
Context Routing Strategy
When the PM uses /ai-prioritize, I automatically:
1. Check Existing AI Features
Source: context-library/prds/, outputs/prds/, outputs/roadmaps/roadmap-data.json
- Look for AI-powered features already on the roadmap
- Pull confidence estimates from any prior analysis
2. Reference Cost Data
Source: context-library/metrics/, connected analytics MCPs
- Pull actual API costs if available
- Estimate based on model pricing if not
3. Check Eval Status
Source: context-library/research/, test datasets if they exist
- Identify which features have golden datasets
- Flag features without any eval capability
4. Apply Company-Specific Context
Source: context-library/business-info-template.md, context-library/research/
- Apply any elevated fallback risk (e.g., financial, medical, legal domains)
- Consider any regulatory or compliance requirements
- Reference your AI feature library for context
5. Cross-Reference with LNO
Source: /prioritize output if available
- If the PM has already classified tasks with LNO, layer AI prioritization on top
- A feature can be "Leverage" in LNO terms but still "Prototype Only" on confidence
Output Quality Self-Check
Before delivering the AI prioritization output, verify:
- Every AI feature scored on all 4 lenses: No feature left without Confidence, Cost, Eval, and Fallback scores
- Decision rules applied consistently: Matrix placement matches the scores
- Catastrophic fallbacks flagged: Any feature where failure is dangerous is explicitly called out
- Cost per user per month calculated: Not just cost per call, but monthly user-level cost
- Eval gaps identified: Features missing eval capability are flagged with "build evals first" recommendation
- Actionable next steps: Each feature has a clear action (ship, MVP test, build eval, prototype, kill)
- No traditional framework assumptions: Output does not assume binary ship/no-ship or fixed impact estimates
Framework credit: Ashutosh Singh (Product Growth). AI Feature Prioritization: Confidence-Gated Framework.