AI Product Management Skills & Decision Frameworks
The user is an AI Product Manager focused on the productization of probabilistic systems — bridging experimental ML capabilities and reliable, scalable software. Apply these frameworks when helping with AI feature scoping, architecture, or reliability decisions.
Product-First Sequencing
Before committing to any technical architecture, validate that users actually need the thing. The workflow is:
- Demo first. Build the smallest possible demonstration to show to real users. The question being answered is: "Do users need this?" — not "Can we build this?" A demo is cheaper than infrastructure and answers the more important question first.
- Validate demand before investing in infrastructure. Only move toward custom hosting, fine-tuning, or specialized retrieval architecture after the application has proven its value. Teams that build infra before validating demand frequently discover they built something well that nobody wanted.
- Simple solutions before complex architectures. When an AI approach is warranted, test the simplest version first — keyword retrieval, intent classifiers, a straightforward prompt — before adding retrieval pipelines, fine-tuned models, or agentic flows. The simple version often works well enough, and the gap it leaves is concrete rather than speculative.
Core Competency: The "Build vs. Buy vs. Rule" Framework
Don't default to AI for every problem. Diagnose the right solution architecture first:
- Heuristics & Rules: Narrow, deterministic problem space (e.g., simple data extraction).
- Traditional ML: Abundant structured data, output probabilistic but bounded (e.g., churn prediction, classification).
- Generative AI (LLMs): Reasoning, synthesis, or unstructured generation (e.g., conversational agents, document summarization).
When the user describes a new feature, explicitly ask: which of these three is the right tool?
Technical Architecture & Execution
- Context Engineering: Optimize context windows for RAG. Use prompt structuring (CoT, ToT) and retrieval strategies (dense vs. sparse) to minimize hallucinations.
- RAG: data preparation before architecture. When a RAG system underperforms, the instinct is to swap the vector database or change the retrieval architecture. The actual bottleneck is almost always data preparation — chunking strategy, metadata tagging, document cleaning, deduplication. Check these before touching architecture. Architecture changes are expensive and rarely the fix.
- Evaluation Design: Build "Ground Truth" datasets and evaluation harnesses. Move beyond "vibe checking" to deterministic metrics (F1, BLEU) and model-based eval (LLM-as-a-Judge).
- Cost-Latency Optimization: Trade off model size, inference speed, and cost. Smaller fine-tuned models for specific tasks; frontier models for general reasoning.
Safety, Governance & Reliability
- Systemic Guardrails: Safety mechanisms outside the model — input/output validation, PII redaction, deterministic business-logic gates — to prevent prompt injection and data leakage.
- Graceful Degradation: UX patterns that handle model failures or high uncertainty without breaking user trust.
Prototyping Stack
If ~/bettersense-work-reflections/profile.md exists, read it and use the stack specified there when writing any prototype code. If no stack is specified in the profile, ask the user before writing code rather than assuming a default.
1---2name: ai-pm-frameworks3description: Use when scoping AI features, deciding build-vs-buy-vs-rule, designing eval harnesses, choosing models for cost/latency trade-offs, or planning guardrails for LLM-based products. Trigger when the user is doing AI product management work — feasibility, architecture, or reliability decisions for probabilistic systems.4---56# AI Product Management Skills & Decision Frameworks78The user is an AI Product Manager focused on the productization of probabilistic systems — bridging experimental ML capabilities and reliable, scalable software. Apply these frameworks when helping with AI feature scoping, architecture, or reliability decisions.910## Product-First Sequencing1112Before committing to any technical architecture, validate that users actually need the thing. The workflow is:13141. **Demo first.** Build the smallest possible demonstration to show to real users. The question being answered is: "Do users need this?" — not "Can we build this?" A demo is cheaper than infrastructure and answers the more important question first.152. **Validate demand before investing in infrastructure.** Only move toward custom hosting, fine-tuning, or specialized retrieval architecture after the application has proven its value. Teams that build infra before validating demand frequently discover they built something well that nobody wanted.163. **Simple solutions before complex architectures.** When an AI approach is warranted, test the simplest version first — keyword retrieval, intent classifiers, a straightforward prompt — before adding retrieval pipelines, fine-tuned models, or agentic flows. The simple version often works well enough, and the gap it leaves is concrete rather than speculative.1718## Core Competency: The "Build vs. Buy vs. Rule" Framework1920Don't default to AI for every problem. Diagnose the right solution architecture first:21- **Heuristics & Rules:** Narrow, deterministic problem space (e.g., simple data extraction).22- **Traditional ML:** Abundant structured data, output probabilistic but bounded (e.g., churn prediction, classification).23- **Generative AI (LLMs):** Reasoning, synthesis, or unstructured generation (e.g., conversational agents, document summarization).2425When the user describes a new feature, explicitly ask: which of these three is the right tool?2627## Technical Architecture & Execution2829- **Context Engineering:** Optimize context windows for RAG. Use prompt structuring (CoT, ToT) and retrieval strategies (dense vs. sparse) to minimize hallucinations.30- **RAG: data preparation before architecture.** When a RAG system underperforms, the instinct is to swap the vector database or change the retrieval architecture. The actual bottleneck is almost always data preparation — chunking strategy, metadata tagging, document cleaning, deduplication. Check these before touching architecture. Architecture changes are expensive and rarely the fix.31- **Evaluation Design:** Build "Ground Truth" datasets and evaluation harnesses. Move beyond "vibe checking" to deterministic metrics (F1, BLEU) and model-based eval (LLM-as-a-Judge).32- **Cost-Latency Optimization:** Trade off model size, inference speed, and cost. Smaller fine-tuned models for specific tasks; frontier models for general reasoning.3334## Safety, Governance & Reliability3536- **Systemic Guardrails:** Safety mechanisms *outside* the model — input/output validation, PII redaction, deterministic business-logic gates — to prevent prompt injection and data leakage.37- **Graceful Degradation:** UX patterns that handle model failures or high uncertainty without breaking user trust.3839## Prototyping Stack4041If `~/bettersense-work-reflections/profile.md` exists, read it and use the stack specified there when writing any prototype code. If no stack is specified in the profile, ask the user before writing code rather than assuming a default.
shwetank