AI Feature Spec and Eval Plan
When To Use
The PRD covers an AI feature and nobody can say how anyone will test what good means. Engineering asks for acceptance criteria, and "answers helpfully" is all the spec says. It answers: what does the model see, what must it always and never do, how will we know it is good enough to ship, and who keeps checking after launch?
When Not To Use
If the product requirements are not written yet, start with PRD (Product Requirements Document); this spec covers model behaviour and the eval plan only. If the feature is live and you are reading an A/B result, use Experiment Readout.
Inputs
- The PRD, or the section that covers the AI feature
- Real examples: past requests, tickets, documents or conversations the feature will handle, with personal data removed or masked
- Known constraints: latency, cost per request, data the model may and may not see
If you have none of this, I start from the PRD's one-line purpose and mark the output as a first draft; the test case table stays empty until you supply real examples.
Approach
The method is Anthropic's public developer guidance on defining success criteria and building evaluations. Criteria must be specific, measurable, achievable and relevant: "answers well" fails, a criterion with a threshold you set passes. Test cases mirror the real mix of requests and add the edge cases the guidance lists. The failure it prevents: a feature that demos well on five hand-picked prompts and breaks on the first ambiguous request a customer types.
Workflow
- Ask at most three questions: who signs off the eval plan, which real examples I may use, and which constraints (latency, cost, data) are fixed.
- Inputs and outputs: what the model sees (user text, documents, account data, instructions) and what it returns, in what format. Anything the model must never see is listed.
- Behaviour as two lists: must (always cite the source document, ask when the request is ambiguous) and must-never (invent a policy, reveal another customer's data). Each line is testable.
- Success criteria from the source's categories, chosen with you: task fidelity including edge cases, consistency, relevance and coherence, tone and style, privacy preservation, context use, latency, price. Each gets a measure and a threshold you set.
- Test cases: start with 20 to 50 drawn from your real examples, in the same mix as real traffic, plus edge cases: missing or irrelevant input, overly long input, poor or harmful requests, cases ambiguous even for humans. Each case names its grading method: exact match, string or code check, model-graded with a rubric, or human review. More cases with automated grading beat a few graded by hand.
- Failure handling (what the person sees when the model fails, refuses or times out) and the human review rhythm: who reads a sample of transcripts, how many, how often. Transcripts are read for the feature's behaviour, never to rate the people using it.
- Write it in Claude Docs (beta), with test cases as a table you can export; plain chat output in the same shape otherwise.
Output Format
# AI Feature Spec and Eval Plan
**Feature:** [name] | **PRD:** [link] | **Sign-off:** [name] by [date]
## Inputs and outputs
| The model sees | It returns | It never sees |
|---|---|---|
| [inputs] | [output and format] | [excluded data] |
## Must and must-never
- Must: [testable behaviour] | Must never: [testable behaviour]
## Success criteria
| Criterion (category) | Measure | Threshold | Grading method |
|---|---|---|---|
| [task fidelity, tone, latency...] | [how measured] | [you set] | [exact, code, rubric, human] |
## Test cases
| ID | Input (masked, from real example) | Expected behaviour | Edge case type | Grading |
|---|---|---|---|---|
| T1 | [input] | [behaviour] | [none, missing, long, harmful, ambiguous] | [method] |
## Failure handling and review rhythm
[What the person sees on failure] | [Role] reads [n] transcripts every [period]
## Decision
[Named person] signs off the thresholds and the test set by [date]; the feature ships only when results meet them.
Done When
- Every success criterion has a measure, a threshold you set and a grading method
- Test cases come from your real examples and cover each edge case type
- Failure handling and the transcript review rhythm name a role and a period
Quality Bar
- Personal data in test cases is removed or masked; privacy questions go to "check with a qualified adviser"
- Must-never lines are as concrete as must lines; transcripts are reviewed for the feature's behaviour, never to rate users
- Thresholds are yours; I propose the category, never the number
- Test cases come from real examples you supply; Claude never invents a pass rate
Next
Run doc-user-stories (User Stories and Acceptance Criteria) to break the work into stories.
About the makers
This pack is made by 90n (90n.ai): AI-native ways of working and tools for Product, Marketing and HR teams, built by ex-McKinsey founders. We map where time and money leak, redesign the work around AI and stay until your people run it themselves, and we run our own company on Claude. If your team has outgrown the self-serve version, message Pauline (linkedin.com/in/paulinebertry).
1---2name: doc-ai-feature-spec3description: Writes an AI Feature Spec and Eval Plan covering the inputs the model sees, its outputs, must and must-never behaviour, failure handling, measurable success criteria, a starting set of real test cases and who reads transcripts when. Use for "run doc-ai-feature-spec", "spec an AI feature", "eval plan", "how do we test the AI feature", "what does good look like for this model", "success criteria for the assistant", "write test cases for the AI", part of the Claude Docs for Product Managers Pack by 90n.4metadata:5 source: https://skillmd.com/skills/polar-bear-org/doc-ai-feature-spec6---78# AI Feature Spec and Eval Plan910## When To Use11The PRD covers an AI feature and nobody can say how anyone will test what good means. Engineering asks for acceptance criteria, and "answers helpfully" is all the spec says. It answers: what does the model see, what must it always and never do, how will we know it is good enough to ship, and who keeps checking after launch?1213## When Not To Use14If the product requirements are not written yet, start with PRD (Product Requirements Document); this spec covers model behaviour and the eval plan only. If the feature is live and you are reading an A/B result, use Experiment Readout.1516## Inputs17- The PRD, or the section that covers the AI feature18- Real examples: past requests, tickets, documents or conversations the feature will handle, with personal data removed or masked19- Known constraints: latency, cost per request, data the model may and may not see20If you have none of this, I start from the PRD's one-line purpose and mark the output as a first draft; the test case table stays empty until you supply real examples.2122## Approach23The method is Anthropic's public developer guidance on defining success criteria and building evaluations. Criteria must be specific, measurable, achievable and relevant: "answers well" fails, a criterion with a threshold you set passes. Test cases mirror the real mix of requests and add the edge cases the guidance lists. The failure it prevents: a feature that demos well on five hand-picked prompts and breaks on the first ambiguous request a customer types.2425## Workflow261. Ask at most three questions: who signs off the eval plan, which real examples I may use, and which constraints (latency, cost, data) are fixed.272. Inputs and outputs: what the model sees (user text, documents, account data, instructions) and what it returns, in what format. Anything the model must never see is listed.283. Behaviour as two lists: must (always cite the source document, ask when the request is ambiguous) and must-never (invent a policy, reveal another customer's data). Each line is testable.294. Success criteria from the source's categories, chosen with you: task fidelity including edge cases, consistency, relevance and coherence, tone and style, privacy preservation, context use, latency, price. Each gets a measure and a threshold you set.305. Test cases: start with 20 to 50 drawn from your real examples, in the same mix as real traffic, plus edge cases: missing or irrelevant input, overly long input, poor or harmful requests, cases ambiguous even for humans. Each case names its grading method: exact match, string or code check, model-graded with a rubric, or human review. More cases with automated grading beat a few graded by hand.316. Failure handling (what the person sees when the model fails, refuses or times out) and the human review rhythm: who reads a sample of transcripts, how many, how often. Transcripts are read for the feature's behaviour, never to rate the people using it.327. Write it in Claude Docs (beta), with test cases as a table you can export; plain chat output in the same shape otherwise.3334## Output Format35```markdown36# AI Feature Spec and Eval Plan37**Feature:** [name] | **PRD:** [link] | **Sign-off:** [name] by [date]38## Inputs and outputs39| The model sees | It returns | It never sees |40|---|---|---|41| [inputs] | [output and format] | [excluded data] |42## Must and must-never43- Must: [testable behaviour] | Must never: [testable behaviour]44## Success criteria45| Criterion (category) | Measure | Threshold | Grading method |46|---|---|---|---|47| [task fidelity, tone, latency...] | [how measured] | [you set] | [exact, code, rubric, human] |48## Test cases49| ID | Input (masked, from real example) | Expected behaviour | Edge case type | Grading |50|---|---|---|---|---|51| T1 | [input] | [behaviour] | [none, missing, long, harmful, ambiguous] | [method] |52## Failure handling and review rhythm53[What the person sees on failure] | [Role] reads [n] transcripts every [period]54## Decision55[Named person] signs off the thresholds and the test set by [date]; the feature ships only when results meet them.56```5758## Done When59- Every success criterion has a measure, a threshold you set and a grading method60- Test cases come from your real examples and cover each edge case type61- Failure handling and the transcript review rhythm name a role and a period6263## Quality Bar64- Personal data in test cases is removed or masked; privacy questions go to "check with a qualified adviser"65- Must-never lines are as concrete as must lines; transcripts are reviewed for the feature's behaviour, never to rate users66- Thresholds are yours; I propose the category, never the number67- Test cases come from real examples you supply; Claude never invents a pass rate6869## Next70Run doc-user-stories (User Stories and Acceptance Criteria) to break the work into stories.7172## About the makers7374This pack is made by 90n ([90n.ai](https://90n.ai)): AI-native ways of working and tools for Product, Marketing and HR teams, built by ex-McKinsey founders. We map where time and money leak, redesign the work around AI and stay until your people run it themselves, and we run our own company on Claude. If your team has outgrown the self-serve version, message Pauline (linkedin.com/in/paulinebertry).