qa-aman Pilot Debrief
Synthesize unstructured pilot data (observations, tickets, feedback transcripts) into a structured debrief report with spec change recommendations. Converts field findings into edge case stories, negative ACs, and design requirements.
Workflow
- Gather pilot inputs — Read all available pilot data sources
- Categorize findings — Sort into UX issues, bugs, feature requests, positive signals
- Map to spec sections — Link each finding to the spec section it affects
- Generate artifacts — Edge case stories, negative ACs, design requirements
- Write debrief report — Structured report for stakeholders
- Output spec change list — Specific edits to make to existing specs
Categorization Framework
1. UX Issues
Problems with how users interact with the product.
- Navigation confusion, unclear feedback, unexpected behavior
- Accessibility gaps, language/comprehension issues Maps to: Design sections (interaction patterns, states, accessibility)
2. Bugs
Things that broke or didn't work as specified.
- Crashes, data loss, wrong behavior, performance issues Maps to: Error States, NFRs
3. Feature Requests
Things users wished existed.
- Direct requests from users, observer inferences from user behavior Maps to: Scope (in-scope for next phase or out-of-scope), backlog
4. Positive Signals
Things that worked well — validates assumptions and celebrates wins.
- Engagement, comprehension, adoption signals Maps to: Success Metrics, Objectives
Synthesis Principles
Observation -> Edge Case User Story
Every unexpected user behavior becomes an edge case story.
Bug -> Negative Acceptance Criteria
Every pilot bug becomes a "must NOT" criterion on the relevant story.
UX Confusion -> Design Requirement
Every navigation/comprehension issue becomes an explicit interaction pattern.
Analytics Gap -> Metric Requirement
Missing data that would have been useful during the pilot.
Output Format
# Pilot Debrief Report
**Pilot:** {name}
**Dates:** {start} - {end}
**Location:** {location}
**Participants:** {count and types}
**Prepared:** {DD-MM-YYYY}
## 1. Executive Summary
{2-3 paragraphs: key findings, overall assessment, top 3 actions}
## 2. Pilot Goals vs Outcomes
| # | Goal | Success Criteria | Outcome | Status |
|---|------|-----------------|---------|--------|
| 1 | {goal} | {criteria} | {actual} | Pass/Partial/Fail |
## 3. Findings by Category
### 3.1 UX Issues ({count})
| # | Finding | Severity | Users Affected | Spec Section | Action |
|---|---------|----------|---------------|-------------|--------|
### 3.2 Bugs ({count})
| # | Finding | Ticket | Severity | Reproducible? | Action |
|---|---------|--------|----------|--------------|--------|
### 3.3 Feature Requests ({count})
| # | Finding | Requested By | Frequency | Recommendation |
|---|---------|-------------|-----------|----------------|
### 3.4 Positive Signals ({count})
| # | Finding | Evidence | Validates |
|---|---------|---------|-----------|
## 4. Generated Artifacts
### 4.1 New Edge Case Stories
### 4.2 New Negative Acceptance Criteria
### 4.3 Design Requirements Updates
### 4.4 New Metrics Required
## 5. Spec Change Recommendations
### Priority: Must Update Before Next Pilot
### Priority: Update Before Next Sprint
### Priority: Backlog
Rules
- Every finding gets categorized — No finding should be left uncategorized
- Every bug becomes an AC — Pilot bugs are the most valuable input for acceptance criteria
- Quantify when possible — "3 out of 5 users" is better than "some users"
- Quote users directly — Direct quotes add credibility and empathy
- Don't speculate on fixes — The debrief identifies what needs to change, not how
- Positive signals matter — Don't only report problems
Quality Checklist
- All findings categorized
- Every bug has a corresponding negative AC
- Quantified observations where possible
- Direct user quotes included
- Spec change recommendations prioritized
- Positive signals documented alongside issues