Experiment readout
Read out the finished test in "$ARGUMENTS" and make a call that can be trusted. The checks run in a fixed order, and a failure early in the order overrides everything after it: a significant p-value means nothing if the test is broken.
Compute with code. Perform every calculation (totals, balances, ratios, variances, projections) with a script — Bash, Python or a spreadsheet formula written to a file — and include the computation alongside the result. Never do arithmetic mentally.
Prerequisites
The results: a file path or a pasted table with, per arm, the users assigned and the outcome (conversions, or a mean and spread). When $ARGUMENTS or the request names a project, use that name as <project> whether or not its folder exists yet in .singularskills/analytics/ (create it when you write). With no project named: with exactly one folder there, use it; with several, ask which; with no folder at all, ask for a project name.
Read the test plan from the ## A/B Test: section of .singularskills/marketing/<project>/03-convert.md when it exists: the planned split, primary metric, guardrails, planned sample size per arm and duration. Without a plan, ask once for those; with no answer, the planned split is unknown: say so, treat the sample size and duration as unknown, and never call the test invalid on an assumed split. If the file has more than one ## A/B Test: section, use the one whose name matches the test in the request or the results file; with several and no match, ask which.
Append this line to .singularskills/log.jsonl before you write the readout (create the folder and file if missing): {"ts":"<ISO timestamp>","skill":"analytics-experiment-readout","workflow":"analytics","phase":null,"project":"<project>","event":"started","artifact":"experiments/<date>-<slug>.md","note":""}, where <date> is today as YYYY-MM-DD and <slug> is two to four words naming the test, never a person.
Process
Check the sample ratio first. Compare the users in each arm with the planned split from the plan (not an assumed 50/50 when the plan says otherwise), using a chi-square goodness-of-fit test computed with a script, as references/experiment-readout.md shows. At p < 0.001 there is a sample ratio mismatch: the assignment or the logging is broken, the result cannot be trusted until the cause is found, and the call is invalid whatever the primary metric shows. List the usual causes to check: users dropped by a redirect or a crash in one arm, bots, a logging change mid-test, mis-bucketing.
When the planned split is unknown (no plan and no answer), test the counts against 50/50 and against the nearest conventional split they fit (90/10, 80/20, 2:1). If they fit neither, or fit a split other than 50/50, the call is inconclusive: planned split unknown, naming the split the counts fit and asking for the plan; a deviation from an assumed split is never invalid.
Without a way to run code, state no test statistic or p-value. Show the observed users per arm and their shares against the planned split, label the check not computed, and if the shares are plainly off the plan (several points away on thousands of users), the call is inconclusive: suspected sample ratio mismatch until the check is computed. Do not go on to call a winner.
Check the test ran as planned: the planned sample per arm and duration were reached, and nobody stopped it early because it looked significant. An unplanned early stop, or a test short of its planned sample or duration, makes the call inconclusive unless the plan used a sequential method.
Compute the primary metric's effect: absolute and relative difference with a 95% confidence interval, the sample per arm and the p-value (two-proportion z-test for a conversion metric; Welch's t-test or a bootstrap for a mean), with a script written to .singularskills/analytics/<project>/ and included beside the result. Scripts read the user's files where they keep them and write only aggregates (counts, sums, rates and intervals): no row-level data, names, emails or raw IDs go into any file under .singularskills/. Without a way to run code, report the raw rates per arm as not computed and make no significance claim.
Check the guardrails the plan names, each with its change and interval. A significantly worse guardrail means don't ship, and blocks shipping even when the call is inconclusive for another reason: say so beside the call.
Weigh the base rate: most tests do not move their primary metric, so a surprising or very large effect needs a replication run before launch; say so when it applies.
Make the call from this table, top row first:
| Situation |
Call |
| Sample ratio mismatch computed at p < 0.001 |
invalid |
| Shares plainly off the planned split, check not computed |
inconclusive: suspected sample ratio mismatch |
| No plan, and the counts do not fit 50/50 |
inconclusive: planned split unknown |
| Unplanned early stop, or planned sample or duration not reached |
inconclusive |
| A guardrail significantly worse |
don't ship |
| Primary metric's interval entirely above zero, guardrails fine |
ship (replicate first if surprising) |
| Primary metric's interval entirely below zero |
don't ship |
| Interval spans zero |
inconclusive, with the effect sizes it rules out |
Write the readout to .singularskills/analytics/<project>/experiments/<date>-<slug>.md with sections Plan (as read, or assumed), Sample ratio, Ran as planned, Primary metric, Guardrails, Call and Next step.
Output
The readout written to .singularskills/analytics/<project>/experiments/<date>-<slug>.md. Present the call first, then the check that decided it, then the primary metric's effect with its interval or not computed.
After you have written the readout, append the started line of Prerequisites to .singularskills/log.jsonl with "event":"finished": {"ts":"<ISO timestamp>","skill":"analytics-experiment-readout","workflow":"analytics","phase":null,"project":"<project>","event":"finished","artifact":"experiments/<date>-<slug>.md","note":""}.
The readout opens with this frontmatter (agent is the agent you run in, or unknown; this utility has no gate, so phase is null and gate is none):
---
skill: analytics-experiment-readout
workflow: analytics
phase: null
project: <project>
started: <ISO timestamp>
finished: <ISO timestamp>
agent: <the agent you run in, or "unknown">
gate: none
---
The readout ends with ## Decisions and ## Open questions, in this shape:
## Decisions
- <what was chosen> — <why> — <what was rejected>
## Open questions
- <what the next phase or a reviewer must settle>
1---2name: analytics-experiment-readout3description: Read out a finished A/B test — sample ratio check first, then whether it ran as planned, the primary metric's effect and interval, guardrails, and a ship, don't ship, inconclusive or invalid call. Use when a test has finished and the user asks whether the change won and can ship. Not for designing a test; see /marketing-ab-test-setup.4metadata:5 source: https://skillmd.com/skills/bromso/analytics-experiment-readout6 argument-hint: "<results file> [project]"7---89# Experiment readout1011Read out the finished test in "$ARGUMENTS" and make a call that can be trusted. The checks run in a fixed order, and a failure early in the order overrides everything after it: a significant p-value means nothing if the test is broken.1213**Compute with code.** Perform every calculation (totals, balances, ratios, variances, projections) with a script — Bash, Python or a spreadsheet formula written to a file — and include the computation alongside the result. Never do arithmetic mentally.1415## Prerequisites1617The results: a file path or a pasted table with, per arm, the users assigned and the outcome (conversions, or a mean and spread). When `$ARGUMENTS` or the request names a project, use that name as `<project>` whether or not its folder exists yet in `.singularskills/analytics/` (create it when you write). With no project named: with exactly one folder there, use it; with several, ask which; with no folder at all, ask for a project name.1819Read the test plan from the `## A/B Test:` section of `.singularskills/marketing/<project>/03-convert.md` when it exists: the planned split, primary metric, guardrails, planned sample size per arm and duration. Without a plan, ask once for those; with no answer, the planned split is unknown: say so, treat the sample size and duration as unknown, and never call the test invalid on an assumed split. If the file has more than one `## A/B Test:` section, use the one whose name matches the test in the request or the results file; with several and no match, ask which.2021Append this line to `.singularskills/log.jsonl` before you write the readout (create the folder and file if missing): `{"ts":"<ISO timestamp>","skill":"analytics-experiment-readout","workflow":"analytics","phase":null,"project":"<project>","event":"started","artifact":"experiments/<date>-<slug>.md","note":""}`, where `<date>` is today as `YYYY-MM-DD` and `<slug>` is two to four words naming the test, never a person.2223## Process24251. **Check the sample ratio first.** Compare the users in each arm with the **planned split** from the plan (not an assumed 50/50 when the plan says otherwise), using a chi-square goodness-of-fit test computed with a script, as `references/experiment-readout.md` shows. At p < 0.001 there is a sample ratio mismatch: the assignment or the logging is broken, the result cannot be trusted until the cause is found, and the call is **invalid** whatever the primary metric shows. List the usual causes to check: users dropped by a redirect or a crash in one arm, bots, a logging change mid-test, mis-bucketing.2627 **When the planned split is unknown** (no plan and no answer), test the counts against 50/50 and against the nearest conventional split they fit (90/10, 80/20, 2:1). If they fit neither, or fit a split other than 50/50, the call is **inconclusive: planned split unknown**, naming the split the counts fit and asking for the plan; a deviation from an assumed split is never **invalid**.2829 **Without a way to run code**, state no test statistic or p-value. Show the observed users per arm and their shares against the planned split, label the check `not computed`, and if the shares are plainly off the plan (several points away on thousands of users), the call is **inconclusive: suspected sample ratio mismatch** until the check is computed. Do not go on to call a winner.30312. **Check the test ran as planned:** the planned sample per arm and duration were reached, and nobody stopped it early because it looked significant. An unplanned early stop, or a test short of its planned sample or duration, makes the call **inconclusive** unless the plan used a sequential method.32333. **Compute the primary metric's effect:** absolute and relative difference with a 95% confidence interval, the sample per arm and the p-value (two-proportion z-test for a conversion metric; Welch's t-test or a bootstrap for a mean), with a script written to `.singularskills/analytics/<project>/` and included beside the result. Scripts read the user's files where they keep them and write only aggregates (counts, sums, rates and intervals): no row-level data, names, emails or raw IDs go into any file under `.singularskills/`. Without a way to run code, report the raw rates per arm as `not computed` and make no significance claim.34354. **Check the guardrails** the plan names, each with its change and interval. A significantly worse guardrail means **don't ship**, and blocks shipping even when the call is **inconclusive** for another reason: say so beside the call.36375. **Weigh the base rate:** most tests do not move their primary metric, so a surprising or very large effect needs a replication run before launch; say so when it applies.38396. **Make the call** from this table, top row first:4041 | Situation | Call |42 | --- | --- |43 | Sample ratio mismatch computed at p < 0.001 | **invalid** |44 | Shares plainly off the planned split, check not computed | **inconclusive: suspected sample ratio mismatch** |45 | No plan, and the counts do not fit 50/50 | **inconclusive: planned split unknown** |46 | Unplanned early stop, or planned sample or duration not reached | **inconclusive** |47 | A guardrail significantly worse | **don't ship** |48 | Primary metric's interval entirely above zero, guardrails fine | **ship** (replicate first if surprising) |49 | Primary metric's interval entirely below zero | **don't ship** |50 | Interval spans zero | **inconclusive**, with the effect sizes it rules out |51527. **Write the readout** to `.singularskills/analytics/<project>/experiments/<date>-<slug>.md` with sections **Plan** (as read, or assumed), **Sample ratio**, **Ran as planned**, **Primary metric**, **Guardrails**, **Call** and **Next step**.5354## Output5556The readout written to `.singularskills/analytics/<project>/experiments/<date>-<slug>.md`. Present the call first, then the check that decided it, then the primary metric's effect with its interval or `not computed`.5758After you have written the readout, append the `started` line of Prerequisites to `.singularskills/log.jsonl` with `"event":"finished"`: `{"ts":"<ISO timestamp>","skill":"analytics-experiment-readout","workflow":"analytics","phase":null,"project":"<project>","event":"finished","artifact":"experiments/<date>-<slug>.md","note":""}`.5960The readout opens with this frontmatter (`agent` is the agent you run in, or `unknown`; this utility has no gate, so `phase` is `null` and `gate` is `none`):6162```yaml63---64skill: analytics-experiment-readout65workflow: analytics66phase: null67project: <project>68started: <ISO timestamp>69finished: <ISO timestamp>70agent: <the agent you run in, or "unknown">71gate: none72---73```7475The readout ends with `## Decisions` and `## Open questions`, in this shape:7677```markdown78## Decisions79- <what was chosen> — <why> — <what was rejected>8081## Open questions82- <what the next phase or a reviewer must settle>83```