# Cro Experiment Monitor

> Twice-daily experiment lifecycle (6 AM & 6 PM UTC): merge detection, validation, running checks, conclusion detection, ship/kill PR merge tracking. Runs as the second pass of the cro-monitor cron, after backlog_monitor.

- Skill: `warpdotdev/cro-experiment-monitor` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add warpdotdev/cro-experiment-monitor`
- Raw SKILL.md: https://api.skillmd.com/api/skills/warpdotdev/cro-experiment-monitor/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Marketing & Growth
- Author: warpdotdev (https://skillmd.com/u/warpdotdev)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/warpdotdev/cro-experiment-monitor

---


# CRO Experiment Monitor

## Configuration

Read `.warp/skills/cro/references/config.md` for Notion IDs, data access, and status labels.
Read `.warp/skills/cro/references/config-slack.md` for Slack posting config (token, channels, Thread TS protocol).

Skill-specific config:
- **Stats library**: `.warp/skills/cro/experiment_monitor/scripts/experiment.py`
- **Monitoring queries**: `.warp/skills/cro/experiment_monitor/scripts/monitoring_queries.sql`
- **Dashboard script**: `.warp/skills/cro/experiment_monitor/scripts/create_dashboard.py`

### Slack token

Every Slack post in this skill MUST use `$EXPERIMENT_BUD_SLACK_BOT_TOKEN`. Use the canonical Python snippet from `config-slack.md` for every post.

Reference snippet (use for every post below — inject `message_text`, `channel='SLACK_NOTIFICATIONS_CHANNEL_ID'`, and `thread_ts` per the Slack Thread TS protocol in `config-slack.md`):

```python
import json, os, sys, urllib.request

token = os.environ.get('EXPERIMENT_BUD_SLACK_BOT_TOKEN', '').strip()
if not token:
    print("FATAL: EXPERIMENT_BUD_SLACK_BOT_TOKEN is unset. Refusing to post.", file=sys.stderr)
    sys.exit(1)

payload = {'channel': 'SLACK_NOTIFICATIONS_CHANNEL_ID', 'text': message_text}
if thread_ts:
    payload['thread_ts'] = thread_ts

req = urllib.request.Request(
    'https://slack.com/api/chat.postMessage',
    data=json.dumps(payload).encode('utf-8'),
    headers={'Authorization': f'Bearer {token}', 'Content-Type': 'application/json; charset=utf-8'},
)
with urllib.request.urlopen(req) as resp:
    result = json.loads(resp.read().decode('utf-8'))
    if not result.get('ok'):
        print(f"Slack post failed: {result.get('error')}", file=sys.stderr)
        sys.exit(1)
```

## Procedure

### Step 0: Read experiments needing attention

Query the Roadmap & Backlog for experiments in active statuses:

```bash
curl -s -X POST "https://api.notion.com/v1/databases/NOTION_ROADMAP_DB_ID/query" \
  -H "Authorization: Bearer $NOTION_INTEGRATION_TOKEN" \
  -H "Notion-Version: 2022-06-28" \
  -H "Content-Type: application/json" \
  -d '{"filter": {"or": [
    {"property": "Status", "select": {"equals": "4-ready for review"}},
    {"property": "Status", "select": {"equals": "5-live:validating"}},
    {"property": "Status", "select": {"equals": "6-live:running"}},
    {"property": "Status", "select": {"equals": "7-analyzing"}}
  ]}}'
```
Also query Roadmap entries in final statuses (`8-no-ship`, `8-inconclusive`, `9-shipped`) where `Ship/Kill PR` is non-empty. For each, verify that the PR has merged and that production cleanup/promotion has completed. If the PR is not merged, log `Lifecycle inconsistency: final status but Ship/Kill PR unmerged`, post a warning in the experiment Slack thread if one has not already been posted in the last 24 hours, and treat the page as blocked from new experiments. Do not move the row back to `7-analyzing` automatically because that can re-trigger duplicate analysis.

Route each experiment to the correct phase based on its status. Process all experiments in a single run.

For each experiment, also fetch the Page Registry entry to get the **Primary Metric** for query adaptation and the **Owner** select for the Slack @-mention. Resolve `{owner_mention}` from `Owner` per `references/config.md` → Page Owners; if `Owner` is empty, omit the mention from any post that includes one.

Apply `references/config.md` → Experiment Scope before routing phases. For SEO-only entries:
- `4-ready for review` with an unmerged Build PR: set Status → `0-archived`, log the reason, and do not post to Slack.
- `5-live:validating`, `6-live:running`, or `7-analyzing`: skip routine CRO automation, log that the live SEO-only experiment needs human cleanup, and do not post routine Slack updates.

---

## Phase 1: Merge Detection (Status = `4-ready for review`)

Check if the Build PR has been merged:

```bash
gh pr view {{BUILD_PR_URL}} --repo your-org/your-marketing-site --json state,mergedAt
```

**If merged:**
- Update Status → `5-live:validating`
- Set `Start Date` → today (the actual launch date)
- Post to Slack thread (following TS protocol): `🚀 Experiment live at 50/50 — monitoring for 3 hours`

**If unmerged for >24 hours** (compare Start Date or PR creation date):
- Post to Slack thread: `⏳ Build PR awaiting merge — <{pr_url}|PR> {owner_mention}` (omit the trailing space + mention if `Owner` is empty)

**If unmerged <24 hours:** do nothing, check next run.

---

## Phase 2: Validation (Status = `5-live:validating`)

Calculate hours since `Start Date`. If < 3 hours, skip — check next run.

**After 3 full hours**, run health checks:

Run queries V1 and M1 from `monitoring_queries.sql`.

Check:
- Both variants receiving traffic (enrollment counts > 0)
- Split ratio approximately 50/50 (±5% tolerance) once the validation window has at least 100 enrolled visitors. For smaller samples, treat split imbalance as directional only; do not hold a low-volume experiment solely for a noisy split if both variants are receiving traffic.
- Variant URL returns 200
- Control has conversions

When using M1/M3 or custom adaptations of them, pre-aggregate `analytics.website_conversion` to one row per `anonymous_id` + `pageview_date` using boolean `LOGICAL_OR` fields before joining to assignments. Do not count raw joined conversion rows directly, because duplicate conversion rows can overstate conversions and trigger incorrect ship/kill decisions.

**If healthy:**
- Update Status → `6-live:running`
- Post to thread: `✅ Validation passed after 3 hours — experiment running normally`

**If unhealthy:**
- Post to thread with issue details: `⚠️ Validation issue on {experiment_id}: {description}. Holding at validating. {owner_mention}` (omit the trailing space + mention if `Owner` is empty)
- Keep status at `5-live:validating` — next run will re-check

---

## Phase 3: Running (Status = `6-live:running`)

### 3.1 — Query experiment data

Run monitoring queries M1 and M3 from `monitoring_queries.sql`. Adapt queries to use the **Primary Metric** from the Page Registry if it differs from the default `did_download_same_day`.

If adapting these queries, keep the same conversion de-duplication pattern: aggregate `analytics.website_conversion` by `anonymous_id` and `pageview_date` first, then join the aggregated conversion rows to the first assignment date. Count visitors from assignments and conversions from the aggregated boolean fields.

### 3.2 — Health checks

Read `Method` from the Notion entry to determine analysis type.

```bash
python3 .warp/skills/cro/experiment_monitor/scripts/experiment.py monitor \
  --control-n {{CONTROL_N}} \
  --control-conv {{CONTROL_CONVERSIONS}} \
  --variant-n {{VARIANT_N}} \
  --variant-conv {{VARIANT_CONVERSIONS}} \
  --target-n {{TARGET_N_PER_VARIANT}}
```

For Bayesian:
```bash
python3 .warp/skills/cro/experiment_monitor/scripts/experiment.py bayesian \
  --control-n {{CONTROL_N}} \
  --control-conv {{CONTROL_CONVERSIONS}} \
  --variant-n {{VARIANT_N}} \
  --variant-conv {{VARIANT_CONVERSIONS}} \
  --day {{CURRENT_DAY}} \
  --max-days {{MAX_DAYS}}
```

**Only post to Slack if:**
- A health issue is detected (split imbalance, zero conversions, extreme lift)
- The experiment has reached conclusion criteria (transitioning status)
- A severe regression guardrail is triggered (~50% drop sustained 3+ days)

Do NOT post routine status updates on every run.

### 3.3 — Check conclusion criteria

**Frequentist:** complete when both variants reach target sample size.

**Bayesian:** complete when `decision` is `ship` or `kill` (or `max_days` reached).

**If concluded:**
- Update Status → `7-analyzing`
- This triggers the analyze agent via Notion automation → bud_server
- **Do NOT post to Slack.** The analyze agent’s `🟢 Experiment Complete` message will cover the conclusion and result in a single post.

**If not concluded:** update Notion with current progress (enrollment counts) and continue. No Slack post.

---

## Phase 4: Ship/Kill PR Merge Detection (Status = `7-analyzing`)

The analyze skill creates the ship/kill PR and stores the URL in the Roadmap's `Ship/Kill PR` field.

Check if that PR has been merged:

```bash
gh pr view {{SHIP_KILL_PR_URL}} --repo your-org/your-marketing-site --json state,mergedAt
```

**If merged:**
- Read the `Result` field from the Notion entry.
- Verify the production deployment containing the merged ship/kill PR is live before touching production Sanity. If the deployment is not live yet, keep Status at `7-analyzing` and check again next run. This prevents production code from routing to a variant page that has already been deleted.
- Apply the post-merge production Sanity mutations from the ship/kill PR body, scoped only to this experiment's artifacts:
  - `ship`: patch the production control `page` document to the promoted content from the merged markdown, then delete or mark completed the generated variant `page` document and matching active `experiment` document.
  - `loss`, `no-ship`, or `inconclusive`: leave the control page untouched, then delete or mark completed the generated variant `page` document and matching active `experiment` document.
- Verify production Sanity cleanup: no active experiment remains for `{{EXPERIMENT_ID}}`; for `ship`, the control page serves the promoted content; for no-ship/inconclusive, the control page remains unchanged.
- Update Status to the final value based on Result:
  - `win` or `ship` → `9-shipped`
  - `loss` or `no-ship` → `8-no-ship`
  - `inconclusive` → `8-inconclusive`
- Set `End Date` → today.
- **Do NOT post to Slack.** The analyze agent’s `🟢 Experiment Complete` message already reported the result. No additional notification needed unless production Sanity cleanup fails; if cleanup fails, post a warning in the thread with the blocker.

**If no Ship/Kill PR URL yet:** the analyze agent may still be running. Do nothing.

**If PR exists but unmerged:** check how long it's been. If >24 hours, nudge: `⏳ Ship/Kill PR awaiting merge — <{pr_url}|PR> {owner_mention}` (omit the trailing space + mention if `Owner` is empty).

---

## Important Notes

- **No ramp phase.** Experiments start at 50/50 when the build PR merges. Validation is a health check, not a traffic ramp.
- **Ship/kill cleanup waits for deploy.** Do not delete variant pages or disable experiment documents until the production deployment containing the ship/kill PR is live.
- **Slack discipline.** Only post when status transitions or something goes wrong. No per-run noise.
- Follow the **Slack Thread TS protocol** from `config-slack.md` for all posts.
- **Slack token:** only `$EXPERIMENT_BUD_SLACK_BOT_TOKEN`. See `config-slack.md`.
- **CRITICAL: Never push directly to `main`** on `your-org/your-marketing-site`.
- The backlog_monitor (first pass) handles chaining to the next experiment when a page becomes idle. This skill does not chain.
- Exit silently when no experiments need attention.