# Cro Analyze

> Full experiment analysis, report, insight capture, ship/kill PR. Triggered by bud_server webhook when a Roadmap entry's Status changes to 7-analyzing.

- Skill: `warpdotdev/cro-analyze` (Agent Skill)
- Install (CLI): `npx skillmds@latest add warpdotdev/cro-analyze`
- Raw SKILL.md: https://api.skillmd.com/api/skills/warpdotdev/cro-analyze/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Product & Planning
- Author: warpdotdev (https://skillmd.com/u/warpdotdev)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/warpdotdev/cro-analyze

---


# CRO Experiment Analysis

## Configuration

Read `.warp/skills/cro/references/config.md` for Notion IDs, data access, and status labels.
Read `.warp/skills/cro/references/config-slack.md` for Slack posting config (token, channels, Thread TS protocol).

Skill-specific config:
- **Stats library**: `.warp/skills/cro/experiment_monitor/scripts/experiment.py`
- **Monitoring queries**: `.warp/skills/cro/experiment_monitor/scripts/monitoring_queries.sql`
- **Priors**: `.warp/skills/cro/references/priors.yaml`

## Input

bud_server passes these fields from the Notion webhook:

```
NOTION_PAGE_ID: <hypothesis-page-id>
EXPERIMENT_ID: example-hero-copy
PAGE_PATH: /example-page
```

## Procedure

### Step 1: Fetch experiment context

Fetch the full Roadmap entry from Notion. Extract: Experiment Desc, Page, Method, Experiment ID, Build PR, Dash, Start Date, Slack Thread TS, Hypothesis, Change Type, Change Description.

Preflight required credentials before doing any analysis work. `$NOTION_INTEGRATION_TOKEN` must be present for Notion reads/writes, and either `$GOOGLE_OAUTH_TOKEN` or `$METABASE_API_KEY` must be present for experiment data. `$EXPERIMENT_BUD_SLACK_BOT_TOKEN` must be present before Step 6. If any required credential is missing, stop immediately and report that the triggering run is missing required Oz team secrets; do not attempt to work around missing credentials by spawning child agents, using alternate Slack tokens, or switching data sources.

**SEO-only scope guard.** Experiment Bud is CRO-only. Read `references/config.md` → Experiment Scope. If this entry is clearly SEO-only, do not create an analysis report, ship/kill PR, or Slack result. Log that the live SEO-only experiment needs human cleanup and exit. Do not silently mutate production artifacts for SEO-only tests.

Also fetch the Page Registry entry for the page to get **Primary Metric**, **Secondary Metric(s)**, and **Owner**. Resolve `{owner_mention}` from `Owner` per `references/config.md` → Page Owners; if `Owner` is empty, omit the `{owner_mention}` line from the Slack post in Step 6.

### Step 2: Run final analysis

Query experiment data using M1 and M3 from `monitoring_queries.sql`.

When adapting M1/M3 for the page's **Primary Metric**, preserve the conversion de-duplication pattern from `monitoring_queries.sql`: aggregate `analytics.website_conversion` to one row per `anonymous_id` + `pageview_date` with `LOGICAL_OR` metric fields before joining to experiment assignments. Do not count raw joined conversion rows directly, because duplicate conversion rows can materially overstate conversions and produce incorrect ship/kill decisions.

Run the appropriate analysis command based on the `Method` field from the Notion entry:

**Frequentist:**
```bash
python3 .warp/skills/cro/experiment_monitor/scripts/experiment.py significance \
  --control-n {{CONTROL_N}} \
  --control-conv {{CONTROL_CONVERSIONS}} \
  --variant-n {{VARIANT_N}} \
  --variant-conv {{VARIANT_CONVERSIONS}}
```

**Bayesian:**
```bash
python3 .warp/skills/cro/experiment_monitor/scripts/experiment.py bayesian \
  --control-n {{CONTROL_N}} \
  --control-conv {{CONTROL_CONVERSIONS}} \
  --variant-n {{VARIANT_N}} \
  --variant-conv {{VARIANT_CONVERSIONS}} \
  --day {{CURRENT_DAY}} \
  --max-days {{MAX_DAYS}}
```

The script returns a `recommendation` (frequentist) or `decision` (Bayesian) field with the ship/kill outcome. Use that directly — the thresholds and min sample guardrails are implemented in the script.

Also check the M3 paid vs organic breakout. Flag if they move in opposite directions.

### Step 3: Create analysis report

Create a subpage under Reports (see config.md for Reports page ID).

Title: `YYYY-MM-DD {{PAGE_PATH}} {{EXPERIMENT_ID}} Experiment Analysis`

Report structure:
- **Result summary**: ship/no-ship/inconclusive with key numbers
- **Statistical results**: For frequentist: lift, CI, p-value, sample sizes. For Bayesian: P(variant > control), expected uplift, risk of loss, posterior parameters.
- **Segment breakouts**: Paid vs organic, OS/device if relevant
- **Key learning**: What did we learn? What should we test next?

Save the report URL.

### Step 4: Write to Notion databases

**4a. Update Roadmap entry:**
- `Result` → `win`, `loss`, or `inconclusive`
- `Lift` → relative lift value
- `End Date` → today
- `Analysis Link` → report URL

Do **not** update `Status` in this skill. Leave the Roadmap entry at `7-analyzing` after analysis and PR creation. `experiment_monitor` Phase 4 is the only step allowed to move an analyzed experiment to `9-shipped`, `8-no-ship`, or `8-inconclusive`, and only after the ship/kill PR has merged, the production deployment containing it is live, and scoped production Sanity cleanup/promotion has completed. This prevents pages from becoming eligible for the next experiment while code-defined experiment cleanup is still unmerged.

**4b. Create Analysis Log entry:**
- Title: same as report title
- Status: `completed`
- Date: today
- Pages Analyzed: `{{PAGE_PATH}}`
- Full Report: report URL
- Key Takeaways: 1-2 sentence summary

**4c. Create Knowledge Base insight:**
- Insight: `{{EXPERIMENT_ID}} — {result} ({relative_lift}% lift)`
- Insight Source: `experiment`
- Tags: `experiment-result`
- Page: `{{PAGE_PATH}}`
- Analysis Link: report URL
- Summary: key learning from the experiment

### Step 5: Create ship/kill PR

All code and markdown changes go in a single PR against `your-org/your-marketing-site` base branch `main`. Sanity is the runtime source of truth, but do not apply destructive production Sanity mutations before the ship/kill PR has merged and deployed; production may still be running the code-defined experiment until that deployment is live. Use the `[ship-kill]/{{EXPERIMENT_ID}}` branch convention. Do not run broad production pushes.

**If ship (variant wins):**
1. **Remove the experiment:** delete `your-marketing-site/src/lib/ab/experiments/<experiment-id>.experiment.ts`, then run `pnpm run generate:experiments` to regenerate `experiments/index.ts`. The middleware will stop intercepting traffic on next deploy.
2. **Promote the variant content to control.** Copy `src/content/pages/<slug>-v<n>.md`'s frontmatter blocks over `src/content/pages/<slug>.md` (keep the control's `title`, `slug`, and `seoNoindex` — do NOT carry the variant's `seoNoindex: true` over to control). Then delete `src/content/pages/<slug>-v<n>.md`.
3. **Prepare the post-merge production Sanity plan.** The PR body must include the exact control slug, variant slug, experiment ID, and production Sanity actions for `experiment_monitor` Phase 4. Do not patch production Sanity yet. After the ship/kill PR merges and the production deployment containing it is live, `experiment_monitor` will patch the production control page to the promoted content, then delete or mark completed the variant document and active `experiment` document.
4. PR title: `[Ship] {{EXPERIMENT_ID}} — variant wins`.

**If no-ship (control wins or inconclusive):**
1. **Remove the experiment:** delete `your-marketing-site/src/lib/ab/experiments/<experiment-id>.experiment.ts`, then run `pnpm run generate:experiments` to regenerate `experiments/index.ts`.
2. **Delete the variant markdown:** `rm src/content/pages/<slug>-v<n>.md`.
3. **Prepare the post-merge production Sanity cleanup plan.** The PR body must include the exact variant slug and experiment ID for `experiment_monitor` Phase 4. Do not delete production Sanity artifacts yet. After the ship/kill PR merges and the production deployment containing it is live, `experiment_monitor` will delete or mark completed the active `experiment` document and delete the variant `page` document. Control content is untouched.
4. PR title: `[No-Ship] {{EXPERIMENT_ID}} — reverting to control`.

PR body: full analysis results, recommendation, link to Notion entry and report, list of files changed, plus a **Post-merge production Sanity plan** containing the exact control slug, variant slug, experiment ID, and expected cleanup/promotion actions for `experiment_monitor`.

Store the PR URL in the Roadmap entry's `Ship/Kill PR` field.

### Step 6: Post to Slack thread

Follow the **Slack Thread TS protocol** from `config-slack.md`. Post the result to the experiment's thread.

**Token: use ONLY `$EXPERIMENT_BUD_SLACK_BOT_TOKEN`.** Use the canonical snippet from `config-slack.md`:

```python
import json, os, sys, urllib.request

token = os.environ.get('EXPERIMENT_BUD_SLACK_BOT_TOKEN', '').strip()
if not token:
    print("FATAL: EXPERIMENT_BUD_SLACK_BOT_TOKEN is unset. Refusing to post.", file=sys.stderr)
    sys.exit(1)

payload = {'channel': 'SLACK_NOTIFICATIONS_CHANNEL_ID', 'text': message_text}
if thread_ts:
    payload['thread_ts'] = thread_ts

req = urllib.request.Request(
    'https://slack.com/api/chat.postMessage',
    data=json.dumps(payload).encode('utf-8'),
    headers={'Authorization': f'Bearer {token}', 'Content-Type': 'application/json; charset=utf-8'},
)
with urllib.request.urlopen(req) as resp:
    result = json.loads(resp.read().decode('utf-8'))
    if not result.get('ok'):
        print(f"Slack post failed: {result.get('error')}", file=sys.stderr)
        sys.exit(1)
```

Message templates:

Omit the final `{owner_mention}` line entirely if `Owner` is empty.

**Frequentist format:**
```
{emoji} Experiment Complete — {experiment_desc}
{page_path} · {duration} days · {total_visitors} visitors
•  Control: {control_cvr}% · Variant: {variant_cvr}%
•  {relative_lift}% lift · p={p_value}
{one_line_learning}
🔗 <{pr_url}|Ship/Kill PR> · <{report_url}|Report>
{owner_mention}
```

**Bayesian format:**
```
{emoji} Experiment Complete — {experiment_desc}
{page_path} · {duration} days · {total_visitors} visitors
•  Control: {control_cvr}% · Variant: {variant_cvr}%
•  P(variant > control): {prob}% · Expected uplift: {uplift}%
{one_line_learning}
🔗 <{pr_url}|Ship/Kill PR> · <{report_url}|Report>
{owner_mention}
```

Use 🟢 for ship, 🔴 for no-ship, 🟡 for inconclusive.

## Important Notes

- **CRITICAL: Never push directly to `main`** on `your-org/your-marketing-site`. Ship/kill changes go through PRs.
- **CRITICAL: Never set final Roadmap status from this skill.** `cro-analyze` records the result, report, and ship/kill PR but must leave `Status = 7-analyzing`. Final status is owned by `experiment_monitor` after PR merge, production deploy, and scoped Sanity cleanup.
- **Sanity is the runtime source of truth.** Removing an experiment file from `experiments/` and the markdown variant file is necessary but not sufficient — the production Sanity `experiment` doc and affected `page` docs must also reflect the new state, or users will see stale variants.
- **Do not mutate production Sanity before the ship/kill deploy is live.** The code-defined experiment can keep routing until production has deployed the removal PR. Early deletion of the variant page can break assigned traffic.
- **Experiment Bud production-write exception.** General agents must not write production Sanity. Experiment Bud may write production Sanity only for the current experiment's control page, generated variant page, and matching experiment document, and only in the launch or post-merge ship/kill phases described here.
- The experiment_monitor skill's Phase 4 detects when the ship/kill PR is merged and updates the final status.
- Always cite specific numbers in the report and Slack message. No vague claims.
- Print the full analysis to the agent session output as well.