Outcome Review

Outcome Review

Query analytics for a shipped feature, synthesize an outcome report, and optionally create follow-up Jira work. Entered independently after shipping — days or weeks later.

Step 1: Detect Available Tools

Check which MCP tools are available:

Tier 1 — PostHog MCP: If you have access to query-run, get-experiment, list-experiments, get-feature-flag, or create-annotation as MCP tools, use Tier 1.

Tier 2 — Manual Metrics: If no PostHog MCP tools are available, ask the user to provide metrics directly:

"I don't have PostHog MCP access. Please share any of the following:

• Dashboard screenshots or metric summaries
• Adoption numbers, funnel data, or error rates
• Experiment results if applicable
• Any specific concerns about the shipped feature"

Step 2: Identify the Feature

1. Check for a learn baseline file in ~/.claude/.skill-learn-baselines/:
   • List files, match by feature name or branch name from the user's prompt
   • If found, use the baseline's shipped_at, ship_method, hypotheses, and jira_ticket fields
   • If ship_method is "pull_request", verify the PR was actually merged before proceeding (check pr_url via gh pr view)
2. If no baseline found, ask the user:

   "Which feature should I review? Please provide the feature name, branch name, or Jira ticket ID."

Step 3: Gather Metrics

Tier 1 (PostHog MCP available):

1. Query adoption metrics via query-run with HogQL:
   • Event counts for the feature's key events since shipped_at
   • Compare to the period before shipping (same duration)
   • If the baseline has non-null hypotheses, use each hypothesis's metric field to target specific events/properties instead of generic adoption queries
2. Check experiment results if applicable:
   • list-experiments to find experiments linked to the feature
   • get-experiment for results, significance, and variant performance
3. Check feature flag status:
   • get-feature-flag for rollout percentage and targeting rules
4. Check error rates:
   • query-run for error events associated with the feature

Tier 2 (Manual):

1. Ask the user to share metrics from their dashboards
   • If the baseline has hypotheses, present each hypothesis and its metric to the user: "For H1 ([description]), I need the current value of [metric]. What is it?"
2. Ask about any observed regressions or improvements
3. Synthesize from what the user provides

Step 4: Synthesize Outcome Report

Present a structured report:

Outcome Report

Feature: [name] | Shipped: [date] | Branch: [name]

Adoption: [metrics summary — event counts, trend direction, comparison to pre-ship baseline]

Quality: [error rates, regression indicators]

Experiments: [results if applicable — significance, winning variant, effect size]

Assessment: One of:

• Positive — Metrics improved, no regressions. Close the loop.
• Regression detected — [specific metric] degraded by [amount]. Investigate.
• Inconclusive — Insufficient data. Revisit in [N] days.
• Mixed — [positive metrics] improved but [negative metrics] regressed. Judgment call.

Hypothesis Validation (when baseline has non-null hypotheses):

Status values:

• Confirmed — Actual meets or exceeds target
• Not confirmed — Actual does not meet target
• Inconclusive — Insufficient data, or validation window has not elapsed
• Partially confirmed — Directionally correct but below target threshold

Cause (required for any non-Confirmed hypothesis) — diagnose WHY before recommending. A low metric does not by itself mean the feature failed; mistaking the cause is how a working feature gets reverted. Classify the cause and let it drive the recommendation:

• instrumentation-broken — the metric pipeline is wrong, not the feature (e.g. event renamed/never fired, 0 events while manual QA confirms the flow works). → Fix tracking and re-measure; do NOT change the feature.
• adoption-gap — the feature works but is under-exposed or undiscovered (e.g. flag at 5% rollout; healthy within the exposed cohort). → Increase rollout/discoverability, then re-measure before judging the feature.
• product-miss — fully exposed and correctly measured, users still don't convert. → Iterate on or roll back the feature.
• inconclusive-data — not enough signal yet. → Extend the window; do not act.

Add the chosen cause to the row (or list it under the table) for each non-Confirmed hypothesis (this includes Partially confirmed and Inconclusive, not only Not confirmed), and make the recommendation follow from it.

When hypotheses is null in the baseline (or no baseline found): skip this section entirely. Fall back to the existing generic metrics flow with no behavioral change.

Recommendations: Specific next actions based on the assessment — and, for each non-Confirmed hypothesis, consistent with its diagnosed Cause above.

Step 5: User Decision Gate

Present the report and ask:

"Based on this outcome review, would you like me to:

1. Close the loop — no follow-up needed
2. Create follow-up Jira tickets — I'll draft tickets for the recommended actions (requires your approval before creation)
3. Investigate further — dig deeper into a specific metric or regression"

Wait for the user's choice.

Step 6: Follow-Up Actions

If "Create follow-up tickets" (and Atlassian Rovo MCP available):

1. If the baseline lacks jira_ticket or the feature's parent ticket is unknown, find it first: call search(cloudId, "<feature name>") — the Rovo cross-system search returns the original ticket alongside any linked Confluence docs in one call. Fall back to searchJiraIssuesUsingJql only if search returns no matches.
2. Draft the ticket(s) — title, description, acceptance criteria, priority.
3. Present each draft to the user for approval.
4. Only after explicit approval: createJiraIssue to create the ticket.
5. addCommentToJiraIssue on the original ticket with the outcome summary.

If Atlassian Rovo MCP unavailable:

"I don't have Atlassian Rovo MCP access. Here are the recommended follow-up tickets — please create them manually: [formatted ticket descriptions]"

Step 7: Transition

If follow-up work was identified:

"If follow-up work is needed, invoke Skill(auto-claude-skills:product-discovery) or Skill(superpowers:brainstorming) to begin the next cycle."

If the loop is closed:

"Outcome review complete. The feature loop is closed."