Read the Experiment Result
When To Use
The test ended and everyone reads the dashboard differently. Use it when you say "read the results of the pricing page test" and want one reading held against the rule you set before launch. It answers: is the data clean, did the primary metric move beyond noise, did a guardrail break, and what does the pre-set rule say to do? It runs in any Claude chat, from an experiment-data connector (read only) or an exported file analysed with code.
When Not To Use
If there was no control group, this is not an experiment: use Analyse the Usage Data and call it observational. If the test is still running, do not read it early; peeking inflates false wins, so wait for the planned end.
Inputs
- The test plan: hypothesis, primary metric, guardrails, planned split, planned duration and the decision rule
- The results by variant: assignment counts, metric values, intervals if the tool gives them (export or connector)
- Any change made mid-test (metric, traffic, code)
If you have none of this, I start from the dashboard export and mark the output as a first draft, with the missing pre-set rule stated on line one.
Approach
Online controlled experiments as described by Kohavi, Longbotham, Sommerfield and Henne (2009): decide the evaluation criterion in advance and read it against that. Sample ratio mismatch (Fabijan et al., KDD 2019) comes first: when the observed split differs from the planned one, the data has a quality problem that can make a loss look like a win. The judgment: the rule was written before the result, and it does not move now. The failure it prevents: a flat test rescued by the one segment that happened to go up.
Workflow
- Ask up to three questions: what was the decision rule before launch, what changed during the test, and who makes the ship call?
- Sample ratio check first: observed split against planned, with a chi-square test. A mismatch past the threshold the user sets stops the readout until it is explained (bot traffic, a redirect, a tracking bug).
- Restate the hypothesis and the pre-set decision rule word for word. If none was set, say so on line one: the readout becomes a learning note, not a verdict.
- Primary metric: the effect with its interval, not just a point. An interval that spans zero is "no detectable effect", not "slightly positive".
- Guardrails next: any guardrail past its limit overrides a primary-metric win.
- Segments marked exploratory: a pattern in one segment is a hypothesis for the next test, never a rescue for a flat result. Flag peeking, early stops or changed metrics.
- Apply the rule: ship, iterate or stop. Where the team disagrees with the rule's answer, record what they believe and why, separately.
Output Format
# Experiment Readout
Test: [name] | Dates: [start to end] | Planned split: [ratio] | Decider: [role]
## Data check
| Check | Planned | Observed | Result |
|---|---|---|---|
| Sample ratio | [ratio] | [counts] | [pass / mismatch, p = value] |
| Mid-test changes | none | [list] | [clean / flagged] |
## Hypothesis and rule
- Hypothesis: [as written before launch]
- Rule: [ship if, iterate if, stop if, as written before launch, or "none set"]
## Results
| Metric | Type | Control | Variant | Effect and interval | Limit | Reading |
|---|---|---|---|---|---|---|
| [metric] | Primary | [value] | [value] | [effect, interval] | n/a | [moved / no detectable effect] |
| [metric] | Guardrail | [value] | [value] | [effect, interval] | [limit] | [held / broken] |
## Exploratory segments
- [segment]: [pattern], a hypothesis for the next test
## What the rule says
[Ship / iterate / stop], because [reason]. Team view if different: [view and why].
## Decision
[Decider role] makes the ship, iterate or stop call by [date] and records it here.
Done When
- The sample ratio check is done and passed, or the readout is stopped with the reason
- The primary metric is reported with its interval, against the pre-set rule
- Every guardrail has a reading against its limit
- Segments are labelled exploratory, and mid-test changes are listed
Quality Bar
- Thresholds do not move after results arrive. No invented values, intervals or significance; a missing measure makes the test unreadable, and the readout says so.
- Low-traffic tests may not answer the question; say so rather than stretch the reading.
- Results in aggregate by variant; no per-user results.
- Claude reads the numbers; you make the ship call.
Next
Run pmc-write-executive-summary (Write the Executive Summary) to tell leadership the result.
About the makers
This pack is made by 90n (90n.ai): AI-native ways of working and tools for Product, Marketing and HR teams, built by ex-McKinsey founders. We map where time and money leak, redesign the work around AI and stay until your people run it themselves, and we run our own company on Claude. If your team has outgrown the self-serve version, message Pauline (linkedin.com/in/paulinebertry).
1---2name: pmc-read-the-experiment3description: Reads a finished A/B test into an Experiment Readout with the hypothesis, a sample ratio check, the primary metric with its interval, guardrails, exploratory segments and a ship, iterate or stop call against the rule set before launch. Use for "run pmc-read-the-experiment", "read the results of the pricing page test", "check the sample ratio before anything else", "ship, iterate or stop", "is this test significant", "everyone reads the dashboard differently", "A/B test readout", part of the Claude Opus 5.5 for Product Managers Pack by 90n.4metadata:5 source: https://skillmd.com/skills/polar-bear-org/pmc-read-the-experiment6---78# Read the Experiment Result910## When To Use11The test ended and everyone reads the dashboard differently. Use it when you say "read the results of the pricing page test" and want one reading held against the rule you set before launch. It answers: is the data clean, did the primary metric move beyond noise, did a guardrail break, and what does the pre-set rule say to do? It runs in any Claude chat, from an experiment-data connector (read only) or an exported file analysed with code.1213## When Not To Use14If there was no control group, this is not an experiment: use Analyse the Usage Data and call it observational. If the test is still running, do not read it early; peeking inflates false wins, so wait for the planned end.1516## Inputs17- The test plan: hypothesis, primary metric, guardrails, planned split, planned duration and the decision rule18- The results by variant: assignment counts, metric values, intervals if the tool gives them (export or connector)19- Any change made mid-test (metric, traffic, code)20If you have none of this, I start from the dashboard export and mark the output as a first draft, with the missing pre-set rule stated on line one.2122## Approach23Online controlled experiments as described by Kohavi, Longbotham, Sommerfield and Henne (2009): decide the evaluation criterion in advance and read it against that. Sample ratio mismatch (Fabijan et al., KDD 2019) comes first: when the observed split differs from the planned one, the data has a quality problem that can make a loss look like a win. The judgment: the rule was written before the result, and it does not move now. The failure it prevents: a flat test rescued by the one segment that happened to go up.2425## Workflow261. Ask up to three questions: what was the decision rule before launch, what changed during the test, and who makes the ship call?272. Sample ratio check first: observed split against planned, with a chi-square test. A mismatch past the threshold the user sets stops the readout until it is explained (bot traffic, a redirect, a tracking bug).283. Restate the hypothesis and the pre-set decision rule word for word. If none was set, say so on line one: the readout becomes a learning note, not a verdict.294. Primary metric: the effect with its interval, not just a point. An interval that spans zero is "no detectable effect", not "slightly positive".305. Guardrails next: any guardrail past its limit overrides a primary-metric win.316. Segments marked exploratory: a pattern in one segment is a hypothesis for the next test, never a rescue for a flat result. Flag peeking, early stops or changed metrics.327. Apply the rule: ship, iterate or stop. Where the team disagrees with the rule's answer, record what they believe and why, separately.3334## Output Format35```markdown36# Experiment Readout37Test: [name] | Dates: [start to end] | Planned split: [ratio] | Decider: [role]38## Data check39| Check | Planned | Observed | Result |40|---|---|---|---|41| Sample ratio | [ratio] | [counts] | [pass / mismatch, p = value] |42| Mid-test changes | none | [list] | [clean / flagged] |43## Hypothesis and rule44- Hypothesis: [as written before launch]45- Rule: [ship if, iterate if, stop if, as written before launch, or "none set"]46## Results47| Metric | Type | Control | Variant | Effect and interval | Limit | Reading |48|---|---|---|---|---|---|---|49| [metric] | Primary | [value] | [value] | [effect, interval] | n/a | [moved / no detectable effect] |50| [metric] | Guardrail | [value] | [value] | [effect, interval] | [limit] | [held / broken] |51## Exploratory segments52- [segment]: [pattern], a hypothesis for the next test53## What the rule says54[Ship / iterate / stop], because [reason]. Team view if different: [view and why].55## Decision56[Decider role] makes the ship, iterate or stop call by [date] and records it here.57```5859## Done When60- The sample ratio check is done and passed, or the readout is stopped with the reason61- The primary metric is reported with its interval, against the pre-set rule62- Every guardrail has a reading against its limit63- Segments are labelled exploratory, and mid-test changes are listed6465## Quality Bar66- Thresholds do not move after results arrive. No invented values, intervals or significance; a missing measure makes the test unreadable, and the readout says so.67- Low-traffic tests may not answer the question; say so rather than stretch the reading.68- Results in aggregate by variant; no per-user results.69- Claude reads the numbers; you make the ship call.7071## Next72Run pmc-write-executive-summary (Write the Executive Summary) to tell leadership the result.7374## About the makers7576This pack is made by 90n ([90n.ai](https://90n.ai)): AI-native ways of working and tools for Product, Marketing and HR teams, built by ex-McKinsey founders. We map where time and money leak, redesign the work around AI and stay until your people run it themselves, and we run our own company on Claude. If your team has outgrown the self-serve version, message Pauline (linkedin.com/in/paulinebertry).