A/B testing
A test only answers the question it was designed for: a hypothesis written before launch, one primary metric, a sample size and stopping rule fixed in advance, and an analysis that reads the p-value and interval for what they are. This skill is the tool-agnostic method, from [coreyhaines31/marketingskills, ab-testing] (MIT, (c) 2025 Corey Haines, copied with its notice). Its sample-size tables are replaced by values computed from the standard two-proportion formula (see SAMPLE-SIZE.md). Running a test in PostHog (flags, exposures, its statistics engine and health checks) is posthog EXPERIMENTS.md and the files it links. Test ideas for a page are cro SKILL.md; variant copy is anti-slop-copy SKILL.md. Store experiments are aso SKILL.md; paywall tests are app-growth SKILL.md.
Product context: read .agents/product-context.md first if it exists (defined in anti-slop-copy CONTEXT.md) and ask only for what it doesn't cover. The source names this file .agents/product-marketing.md.
Apply only the routes and checks for the selected job. Read a linked skill's named file from its installed folder when available, or from the linked source. Report an unavailable required reference instead of inventing it. [skill-authoring, DISTRIBUTION.md#composing-skills]
| The job |
Read |
| The required inputs, a sample-size table computed from the formula, duration with equal and unequal splits, minimum and maximum duration, multiple variants, too-high samples, sequential testing, calculators, the can-I-run-this check |
SAMPLE-SIZE.md |
| Test plan, results write-up, test repository row, quick brief, stakeholder update, prioritization scorecard, hypothesis bank, playbook entry |
TEMPLATES.md |
| The experiment loop, hypothesis sources, ICE scoring, velocity, cadence, common mistakes, the questions to ask |
PROGRAM.md |
| Running it in PostHog: plan, rollout and split, metrics, running time, statistics, health checks |
posthog EXPERIMENTS.md |
| Screening headlines in ads, then proving the winner on the landing page with a fresh randomized test |
paid-ads CAMPAIGN-PLANNING.md |
Before designing
[coreyhaines31/marketingskills, ab-testing] Understand three things first:
- Test context: what are you trying to improve, and what change are you considering?
- Current state: baseline conversion rate and current traffic volume.
- Constraints: technical complexity, timeline, tools available.
The hypothesis
[coreyhaines31/marketingskills, ab-testing] Not "let's see what happens": a specific prediction, based on reasoning or data.
Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].
- Weak: "Changing the button color might increase clicks."
- Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
- [coreyhaines31/marketingskills, ad-creative] For a creative test the hypothesis names a metric, a comparison (the control, the audience) and a direction, plus a magnitude when a baseline exists: "beats the control on CTR among renters". Ad creative itself is ad-creative SKILL.md.
Test types
[coreyhaines31/marketingskills, ab-testing]
| Type |
Description |
Traffic needed |
| A/B |
Two versions, single change |
Moderate |
| A/B/n |
Multiple variants |
Higher |
| MVT |
Multiple changes in combinations |
Very high |
| Split URL |
Different URLs for variants |
Moderate |
- MVT combinations multiply: 2 headlines x 2 images x 2 CTAs = 8 combinations. When traffic can't support that, run sequential A/B tests instead.
Metrics
[coreyhaines31/marketingskills, ab-testing]
- Primary: the single metric that matters most, directly tied to the hypothesis; what you use to call the test.
- Secondary: support the primary's interpretation; explain why or how the change worked.
- Guardrail: things that shouldn't get worse; stop the test if one is significantly negative. PostHog's docs call these "counter metrics" (posthog EXPERIMENTS.md).
- Example, pricing page test: primary plan selection rate; secondary time on page and plan distribution; guardrail support tickets and refund rate.
- Example, longer signup form (company size and role added): primary form completion rate; secondary lead quality (SQL conversion rate, activation after signup); guardrail total signup volume. The quantity and quality trade off, and the downstream metrics need a longer observation window.
- [coreyhaines31/marketingskills, marketing-loops] For pricing-page tests: "Judge winners on revenue per visitor, plan mix, refunds, downgrades, churn, and support load - not conversion rate alone." "Do not promote a variant that lifts conversion but lowers revenue-per-visitor or raises refunds/churn."
For opt-in elements, define an equivalent eligibility or exposure point in both randomized arms before the changed behavior. Including people who could never encounter the element dilutes its measured effect; filtering on a click the variant can change instead selects different populations. Sequentially shipping two versions without randomization provides a directional comparison, not a controlled A/B result. PostHog implementation stays in its exposure owner. [coreyhaines31/marketingskills, ab-testing/SKILL.md, Opt-In Elements; https://posthog.com/docs/experiments/exposures]
Traffic allocation
- [coreyhaines31/marketingskills, ab-testing] Standard 50/50 is the default for A/B; conservative 90/10 or 80/20 limits the risk of a bad variant; ramping starts small and increases, for technical risk. Users see the same variant on return; balance exposure across time of day and week.
- PostHog, ../../monitoring/posthog/EXPERIMENTS.md "Always default to an equal split unless the user explicitly requests otherwise", and limit risk by reducing the overall rollout instead: in PostHog an uneven split with the default "Exclude from analysis" can bias the smaller variant. In PostHog, follow PostHog's rule.
- [coreyhaines31/marketingskills, ab-testing] Client-side changes the page after load: quick, can flicker. Server-side picks the variant before render: no flicker, needs dev work. PostHog's flicker fixes are posthog FLAG-EVALUATION.md.
Running the test
[coreyhaines31/marketingskills, ab-testing]
- Pre-launch: hypothesis documented, primary metric defined, sample size calculated, variants implemented correctly, tracking verified, QA completed on all variants. The full list is in the test plan (TEMPLATES.md).
- During: monitor for technical issues, check segment quality, document external factors. Don't peek at results and stop early, change variants, or add traffic from new sources.
- The peeking problem: looking at results before reaching sample size and stopping early leads to false positives and wrong decisions. Pre-commit to the sample size, or use a sequential method built for repeated looks (SAMPLE-SIZE.md).
- Guardrails during the run: "Don't call winners early - but do stop tests where guardrails are significantly negative."
Reading the result
[coreyhaines31/marketingskills, ab-testing], following the American Statistical Association's p-value statement:
- For a pre-specified fixed-horizon test, compare its p-value with the chosen significance level (commonly α = 0.05).
- A p-value describes how incompatible the data are with the null model: the probability, assuming that model and its assumptions, of a result at least as extreme as the observed one. It is not the probability that the result is random, that the null is true, or that a variant will win again.
- A 95% confidence interval comes from a procedure with 95% coverage over repeated samples under its assumptions. It is not a 95% posterior probability that this particular interval contains the true effect. Report the interval and effect size alongside the p-value.
- Use the planned analysis and stopping rule. Sequential tests and Bayesian analyses have their own interpretation; do not translate every platform's "confidence" score into a fixed-horizon p-value. PostHog's Bayesian "chance to win" is read in posthog EXPERIMENT-SIGNIFICANCE.md.
- Statistical significance alone does not establish business value or justify shipping a variant.
Analysis checklist:
- Reached sample size? If not, the result is preliminary.
- Statistically significant? Check confidence intervals.
- Effect size meaningful? Compare to the MDE; project the impact.
- Secondary metrics consistent? Do they support the primary?
- Guardrail concerns? Did anything get worse?
- Segment differences? Mobile vs desktop, new vs returning.
| Result |
Conclusion |
| Significant winner |
Check effect size, uncertainty and guardrails before implementing |
| Significant loser |
Keep control, learn why |
| No significant difference |
Report the effect interval: it may still allow meaningful benefit and harm, or may exclude effects worth pursuing. Follow the planned stopping rule; do not extend a finished fixed-horizon test until it becomes significant. Plan a new test if needed. |
| Mixed signals |
Check data quality and planned metrics; label unplanned segment findings exploratory and confirm them in a new test |
- Two views on a borderline result. [coreyhaines31/marketingskills, ab-testing] says the row above. PostHog agent skills, ../../monitoring/posthog/EXPERIMENT-METRICS.md says "Borderline: Continue, or end as inconclusive" and "Recommend shipping only when the result is statistically significant", and allows shipping an inconclusive result under three conditions (C8).
- Multiple comparisons. PostHog applies no correction across metrics or variants (C4); Optimizely's Stats Engine controls the false discovery rate with a tiered Benjamini-Hochberg procedure (Optimizely). Bonferroni is the manual correction (SAMPLE-SIZE.md).
The rules
- Write the hypothesis before building anything, in the five-line template, with the metric that will prove it. [coreyhaines31/marketingskills, ab-testing]
- Test one thing, bold enough to make a difference; more variants or an MVT only when traffic supports the multiplied sample. [coreyhaines31/marketingskills, ab-testing]
- Fix the sample size, duration and stopping rule before launch, from the baseline of the specific page and metric; never from the source's old lookup tables. [coreyhaines31/marketingskills, ab-testing; SAMPLE-SIZE.md]
- Name one primary, a few secondary and the guardrails up front; stop on a significantly negative guardrail. [coreyhaines31/marketingskills, ab-testing]
- Don't peek and stop early unless the method is sequential and was turned on before launch. [coreyhaines31/marketingskills, ab-testing; ../../monitoring/posthog/EXPERIMENT-SIGNIFICANCE.md]
- Read p-values and intervals as the ASA statement describes: never "a 96% chance it works" from p = 0.04, never "a 4% chance it was random". [coreyhaines31/marketingskills, ab-testing]
- Don't extend a finished fixed-horizon test until it turns significant; report the interval and plan a new test. [coreyhaines31/marketingskills, ab-testing]
- Unplanned segment findings are exploratory; confirm them in a new test with its own plan. [coreyhaines31/marketingskills, ab-testing]
- Document every test with hypothesis, variants with screenshots, results (sample, metrics, significance), decision and learnings (TEMPLATES.md). [coreyhaines31/marketingskills, ab-testing]
Prove it
From the source's evals: a test plan for "15,000 visitors a month at 3.2%" uses the hypothesis template, names primary, secondary and guardrail metrics, sizes the test from the baseline and warns about peeking; four button colors are flagged as an A/B/n test with a larger sample and a question about whether color is the high-impact change; "3 days in, 95% confidence, call it?" gets the peeking answer, day-of-week effects and sequential testing as the alternative; an MVT request counts its combinations and offers sequential A/B tests when traffic is short; 252/12,000 vs 288/12,000 is computed as p ≈ 0.117 (two-sided pooled z-test), not called significant at 0.05, and not extended; p = 0.04 with a +0.1% to +3.9% interval and a higher refund rate is never told to a CEO as "96% chance it works", and the refund guardrail is weighed before rollout; and a request to write page copy is routed to the copy skill.
1---2name: ab-testing3description: "Designs and reads A/B tests and runs an experimentation program, tool-agnostic: the hypothesis, test types and their traffic, primary, secondary and guardrail metrics, sample size from the standard formula and duration with unequal splits, peeking and sequential testing, reading p-values and intervals the ASA way, inconclusive results, test plans and result write-ups, ICE scoring, velocity and cadence. Use when planning an A/B or multivariate test, asking how long to run one or whether a result is significant, writing a test plan or result, or building an experiment backlog."4metadata:5 source: https://skillmd.com/skills/quegenx/ab-testing6---78# A/B testing910A test only answers the question it was designed for: a hypothesis written before launch, one primary metric, a sample size and stopping rule fixed in advance, and an analysis that reads the p-value and interval for what they are. This skill is the tool-agnostic method, from [coreyhaines31/marketingskills, ab-testing] (MIT, (c) 2025 Corey Haines, copied with its notice). Its sample-size tables are replaced by values computed from the standard two-proportion formula (see [SAMPLE-SIZE.md](SAMPLE-SIZE.md)). Running a test in PostHog (flags, exposures, its statistics engine and health checks) is [posthog EXPERIMENTS.md](https://github.com/Quegenx/agent-skills/blob/main/skills/monitoring/posthog/EXPERIMENTS.md) and the files it links. Test ideas for a page are [cro SKILL.md](https://github.com/Quegenx/agent-skills/blob/main/skills/marketing/cro/SKILL.md); variant copy is [anti-slop-copy SKILL.md](https://github.com/Quegenx/agent-skills/blob/main/skills/anti-ai-slop/anti-slop-copy/SKILL.md). Store experiments are [aso SKILL.md](https://github.com/Quegenx/agent-skills/blob/main/skills/marketing/aso/SKILL.md); paywall tests are [app-growth SKILL.md](https://github.com/Quegenx/agent-skills/blob/main/skills/marketing/app-growth/SKILL.md).1112Product context: read `.agents/product-context.md` first if it exists (defined in [anti-slop-copy CONTEXT.md](https://github.com/Quegenx/agent-skills/blob/main/skills/anti-ai-slop/anti-slop-copy/CONTEXT.md)) and ask only for what it doesn't cover. The source names this file `.agents/product-marketing.md`.1314Apply only the routes and checks for the selected job. Read a linked skill's named file from its installed folder when available, or from the linked source. Report an unavailable required reference instead of inventing it. [skill-authoring, DISTRIBUTION.md#composing-skills]1516| The job | Read |17|---|---|18| The required inputs, a sample-size table computed from the formula, duration with equal and unequal splits, minimum and maximum duration, multiple variants, too-high samples, sequential testing, calculators, the can-I-run-this check | [SAMPLE-SIZE.md](SAMPLE-SIZE.md) |19| Test plan, results write-up, test repository row, quick brief, stakeholder update, prioritization scorecard, hypothesis bank, playbook entry | [TEMPLATES.md](TEMPLATES.md) |20| The experiment loop, hypothesis sources, ICE scoring, velocity, cadence, common mistakes, the questions to ask | [PROGRAM.md](PROGRAM.md) |21| Running it in PostHog: plan, rollout and split, metrics, running time, statistics, health checks | [posthog EXPERIMENTS.md](https://github.com/Quegenx/agent-skills/blob/main/skills/monitoring/posthog/EXPERIMENTS.md) |22| Screening headlines in ads, then proving the winner on the landing page with a fresh randomized test | [paid-ads CAMPAIGN-PLANNING.md](https://github.com/Quegenx/agent-skills/blob/main/skills/marketing/paid-ads/CAMPAIGN-PLANNING.md#landing-page-continuity) |2324## Before designing2526[coreyhaines31/marketingskills, ab-testing] Understand three things first:27281. **Test context:** what are you trying to improve, and what change are you considering?292. **Current state:** baseline conversion rate and current traffic volume.303. **Constraints:** technical complexity, timeline, tools available.3132## The hypothesis3334[coreyhaines31/marketingskills, ab-testing] Not "let's see what happens": a specific prediction, based on reasoning or data.3536```37Because [observation/data],38we believe [change]39will cause [expected outcome]40for [audience].41We'll know this is true when [metrics].42```4344- **Weak:** "Changing the button color might increase clicks."45- **Strong:** "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."46- [coreyhaines31/marketingskills, ad-creative] For a creative test the hypothesis names a metric, a comparison (the control, the audience) and a direction, plus a magnitude when a baseline exists: "beats the control on CTR among renters". Ad creative itself is [ad-creative SKILL.md](https://github.com/Quegenx/agent-skills/blob/main/skills/marketing/ad-creative/SKILL.md).4748## Test types4950[coreyhaines31/marketingskills, ab-testing]5152| Type | Description | Traffic needed |53|------|-------------|----------------|54| A/B | Two versions, single change | Moderate |55| A/B/n | Multiple variants | Higher |56| MVT | Multiple changes in combinations | Very high |57| Split URL | Different URLs for variants | Moderate |5859- **MVT combinations multiply:** 2 headlines x 2 images x 2 CTAs = 8 combinations. When traffic can't support that, run sequential A/B tests instead.6061## Metrics6263[coreyhaines31/marketingskills, ab-testing]6465- **Primary:** the single metric that matters most, directly tied to the hypothesis; what you use to call the test.66- **Secondary:** support the primary's interpretation; explain why or how the change worked.67- **Guardrail:** things that shouldn't get worse; stop the test if one is significantly negative. PostHog's docs call these "counter metrics" ([posthog EXPERIMENTS.md](https://github.com/Quegenx/agent-skills/blob/main/skills/monitoring/posthog/EXPERIMENTS.md#plan-the-experiment)).68- **Example, pricing page test:** primary plan selection rate; secondary time on page and plan distribution; guardrail support tickets and refund rate.69- **Example, longer signup form** (company size and role added): primary form completion rate; secondary lead quality (SQL conversion rate, activation after signup); guardrail total signup volume. The quantity and quality trade off, and the downstream metrics need a longer observation window.70- [coreyhaines31/marketingskills, marketing-loops] For pricing-page tests: "Judge winners on revenue per visitor, plan mix, refunds, downgrades, churn, and support load - not conversion rate alone." "Do not promote a variant that lifts conversion but lowers revenue-per-visitor or raises refunds/churn."7172For opt-in elements, define an equivalent eligibility or exposure point in both randomized arms before the changed behavior. Including people who could never encounter the element dilutes its measured effect; filtering on a click the variant can change instead selects different populations. Sequentially shipping two versions without randomization provides a directional comparison, not a controlled A/B result. PostHog implementation stays in its [exposure owner](https://github.com/Quegenx/agent-skills/blob/main/skills/monitoring/posthog/EXPERIMENTS.md). [coreyhaines31/marketingskills, ab-testing/SKILL.md, Opt-In Elements; https://posthog.com/docs/experiments/exposures]7374## Traffic allocation7576- [coreyhaines31/marketingskills, ab-testing] Standard 50/50 is the default for A/B; conservative 90/10 or 80/20 limits the risk of a bad variant; ramping starts small and increases, for technical risk. Users see the same variant on return; balance exposure across time of day and week.77- [PostHog, ../../monitoring/posthog/EXPERIMENTS.md](https://github.com/Quegenx/agent-skills/blob/main/skills/monitoring/posthog/EXPERIMENTS.md#rollout-and-split) "Always default to an equal split unless the user explicitly requests otherwise", and limit risk by reducing the overall rollout instead: in PostHog an uneven split with the default "Exclude from analysis" can bias the smaller variant. In PostHog, follow PostHog's rule.78- [coreyhaines31/marketingskills, ab-testing] Client-side changes the page after load: quick, can flicker. Server-side picks the variant before render: no flicker, needs dev work. PostHog's flicker fixes are [posthog FLAG-EVALUATION.md](https://github.com/Quegenx/agent-skills/blob/main/skills/monitoring/posthog/FLAG-EVALUATION.md).7980## Running the test8182[coreyhaines31/marketingskills, ab-testing]8384- **Pre-launch:** hypothesis documented, primary metric defined, sample size calculated, variants implemented correctly, tracking verified, QA completed on all variants. The full list is in the test plan ([TEMPLATES.md](TEMPLATES.md#test-plan)).85- **During:** monitor for technical issues, check segment quality, document external factors. Don't peek at results and stop early, change variants, or add traffic from new sources.86- **The peeking problem:** looking at results before reaching sample size and stopping early leads to false positives and wrong decisions. Pre-commit to the sample size, or use a sequential method built for repeated looks ([SAMPLE-SIZE.md](SAMPLE-SIZE.md#sequential-testing)).87- **Guardrails during the run:** "Don't call winners early - but do stop tests where guardrails are significantly negative."8889## Reading the result9091[coreyhaines31/marketingskills, ab-testing], following the [American Statistical Association's p-value statement](https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf):9293- For a pre-specified fixed-horizon test, compare its p-value with the chosen significance level (commonly α = 0.05).94- A p-value describes how incompatible the data are with the null model: the probability, **assuming that model and its assumptions**, of a result at least as extreme as the observed one. It is not the probability that the result is random, that the null is true, or that a variant will win again.95- A 95% confidence interval comes from a procedure with 95% coverage over repeated samples under its assumptions. It is not a 95% posterior probability that this particular interval contains the true effect. Report the interval and effect size alongside the p-value.96- Use the planned analysis and stopping rule. Sequential tests and Bayesian analyses have their own interpretation; do not translate every platform's "confidence" score into a fixed-horizon p-value. PostHog's Bayesian "chance to win" is read in [posthog EXPERIMENT-SIGNIFICANCE.md](https://github.com/Quegenx/agent-skills/blob/main/skills/monitoring/posthog/EXPERIMENT-SIGNIFICANCE.md#c5-bayesian-traps-high).97- Statistical significance alone does not establish business value or justify shipping a variant.9899Analysis checklist:1001011. **Reached sample size?** If not, the result is preliminary.1022. **Statistically significant?** Check confidence intervals.1033. **Effect size meaningful?** Compare to the MDE; project the impact.1044. **Secondary metrics consistent?** Do they support the primary?1055. **Guardrail concerns?** Did anything get worse?1066. **Segment differences?** Mobile vs desktop, new vs returning.107108| Result | Conclusion |109|--------|------------|110| Significant winner | Check effect size, uncertainty and guardrails before implementing |111| Significant loser | Keep control, learn why |112| No significant difference | Report the effect interval: it may still allow meaningful benefit and harm, or may exclude effects worth pursuing. Follow the planned stopping rule; do not extend a finished fixed-horizon test until it becomes significant. Plan a new test if needed. |113| Mixed signals | Check data quality and planned metrics; label unplanned segment findings exploratory and confirm them in a new test |114115- **Two views on a borderline result.** [coreyhaines31/marketingskills, ab-testing] says the row above. [PostHog agent skills, ../../monitoring/posthog/EXPERIMENT-METRICS.md](https://github.com/Quegenx/agent-skills/blob/main/skills/monitoring/posthog/EXPERIMENT-METRICS.md#interpreting-results) says "Borderline: Continue, or end as inconclusive" and "Recommend shipping only when the result is statistically significant", and allows shipping an inconclusive result under three conditions ([C8](https://github.com/Quegenx/agent-skills/blob/main/skills/monitoring/posthog/EXPERIMENT-SIGNIFICANCE.md#c8-shipping-an-inconclusive-result-medium)).116- **Multiple comparisons.** PostHog applies no correction across metrics or variants ([C4](https://github.com/Quegenx/agent-skills/blob/main/skills/monitoring/posthog/EXPERIMENT-SIGNIFICANCE.md#c4-multiple-comparisons-high)); Optimizely's Stats Engine controls the false discovery rate with a tiered Benjamini-Hochberg procedure ([Optimizely](https://docs.optimizely.com/experimentation-strategy/docs/false-discovery-rate-control)). Bonferroni is the manual correction ([SAMPLE-SIZE.md](SAMPLE-SIZE.md#multiple-variants)).117118## The rules1191201. **Write the hypothesis before building anything,** in the five-line template, with the metric that will prove it. [coreyhaines31/marketingskills, ab-testing]1212. **Test one thing,** bold enough to make a difference; more variants or an MVT only when traffic supports the multiplied sample. [coreyhaines31/marketingskills, ab-testing]1223. **Fix the sample size, duration and stopping rule before launch,** from the baseline of the specific page and metric; never from the source's old lookup tables. [coreyhaines31/marketingskills, ab-testing; SAMPLE-SIZE.md]1234. **Name one primary, a few secondary and the guardrails up front;** stop on a significantly negative guardrail. [coreyhaines31/marketingskills, ab-testing]1245. **Don't peek and stop early** unless the method is sequential and was turned on before launch. [coreyhaines31/marketingskills, ab-testing; ../../monitoring/posthog/EXPERIMENT-SIGNIFICANCE.md]1256. **Read p-values and intervals as the ASA statement describes:** never "a 96% chance it works" from p = 0.04, never "a 4% chance it was random". [coreyhaines31/marketingskills, ab-testing]1267. **Don't extend a finished fixed-horizon test until it turns significant;** report the interval and plan a new test. [coreyhaines31/marketingskills, ab-testing]1278. **Unplanned segment findings are exploratory;** confirm them in a new test with its own plan. [coreyhaines31/marketingskills, ab-testing]1289. **Document every test** with hypothesis, variants with screenshots, results (sample, metrics, significance), decision and learnings ([TEMPLATES.md](TEMPLATES.md)). [coreyhaines31/marketingskills, ab-testing]129130## Prove it131132From the source's evals: a test plan for "15,000 visitors a month at 3.2%" uses the hypothesis template, names primary, secondary and guardrail metrics, sizes the test from the baseline and warns about peeking; four button colors are flagged as an A/B/n test with a larger sample and a question about whether color is the high-impact change; "3 days in, 95% confidence, call it?" gets the peeking answer, day-of-week effects and sequential testing as the alternative; an MVT request counts its combinations and offers sequential A/B tests when traffic is short; 252/12,000 vs 288/12,000 is computed as p ≈ 0.117 (two-sided pooled z-test), not called significant at 0.05, and not extended; p = 0.04 with a +0.1% to +3.9% interval and a higher refund rate is never told to a CEO as "96% chance it works", and the refund guardrail is weighed before rollout; and a request to write page copy is routed to the copy skill.
