Cro Experiment Monitor

CRO Experiment Monitor

Configuration

Read .warp/skills/cro/references/config.md for Notion IDs, data access, and status labels. Read .warp/skills/cro/references/config-slack.md for Slack posting config (token, channels, Thread TS protocol).

Skill-specific config:

• Stats library: .warp/skills/cro/experiment_monitor/scripts/experiment.py
• Monitoring queries: .warp/skills/cro/experiment_monitor/scripts/monitoring_queries.sql
• Dashboard script: .warp/skills/cro/experiment_monitor/scripts/create_dashboard.py

Slack token

Every Slack post in this skill MUST use $EXPERIMENT_BUD_SLACK_BOT_TOKEN. Use the canonical Python snippet from config-slack.md for every post.

Reference snippet (use for every post below — inject message_text, channel='SLACK_NOTIFICATIONS_CHANNEL_ID', and thread_ts per the Slack Thread TS protocol in config-slack.md):

import json, os, sys, urllib.request

token = os.environ.get('EXPERIMENT_BUD_SLACK_BOT_TOKEN', '').strip()
if not token:
    print("FATAL: EXPERIMENT_BUD_SLACK_BOT_TOKEN is unset. Refusing to post.", file=sys.stderr)
    sys.exit(1)

payload = {'channel': 'SLACK_NOTIFICATIONS_CHANNEL_ID', 'text': message_text}
if thread_ts:
    payload['thread_ts'] = thread_ts

req = urllib.request.Request(
    'https://slack.com/api/chat.postMessage',
    data=json.dumps(payload).encode('utf-8'),
    headers={'Authorization': f'Bearer {token}', 'Content-Type': 'application/json; charset=utf-8'},
)
with urllib.request.urlopen(req) as resp:
    result = json.loads(resp.read().decode('utf-8'))
    if not result.get('ok'):
        print(f"Slack post failed: {result.get('error')}", file=sys.stderr)
        sys.exit(1)

Procedure

Step 0: Read experiments needing attention

Query the Roadmap & Backlog for experiments in active statuses:

curl -s -X POST "https://api.notion.com/v1/databases/NOTION_ROADMAP_DB_ID/query" \
  -H "Authorization: Bearer $NOTION_INTEGRATION_TOKEN" \
  -H "Notion-Version: 2022-06-28" \
  -H "Content-Type: application/json" \
  -d '{"filter": {"or": [
    {"property": "Status", "select": {"equals": "4-ready for review"}},
    {"property": "Status", "select": {"equals": "5-live:validating"}},
    {"property": "Status", "select": {"equals": "6-live:running"}},
    {"property": "Status", "select": {"equals": "7-analyzing"}}
  ]}}'

Also query Roadmap entries in final statuses (8-no-ship, 8-inconclusive, 9-shipped) where Ship/Kill PR is non-empty. For each, verify that the PR has merged and that production cleanup/promotion has completed. If the PR is not merged, log Lifecycle inconsistency: final status but Ship/Kill PR unmerged, post a warning in the experiment Slack thread if one has not already been posted in the last 24 hours, and treat the page as blocked from new experiments. Do not move the row back to 7-analyzing automatically because that can re-trigger duplicate analysis.

Route each experiment to the correct phase based on its status. Process all experiments in a single run.

For each experiment, also fetch the Page Registry entry to get the Primary Metric for query adaptation and the Owner select for the Slack @-mention. Resolve {owner_mention} from Owner per references/config.md → Page Owners; if Owner is empty, omit the mention from any post that includes one.

Apply references/config.md → Experiment Scope before routing phases. For SEO-only entries:

• 4-ready for review with an unmerged Build PR: set Status → 0-archived, log the reason, and do not post to Slack.
• 5-live:validating, 6-live:running, or 7-analyzing: skip routine CRO automation, log that the live SEO-only experiment needs human cleanup, and do not post routine Slack updates.

Phase 1: Merge Detection (Status = 4-ready for review)

Check if the Build PR has been merged:

gh pr view {{BUILD_PR_URL}} --repo your-org/your-marketing-site --json state,mergedAt

If merged:

• Update Status → 5-live:validating
• Set Start Date → today (the actual launch date)
• Post to Slack thread (following TS protocol): 🚀 Experiment live at 50/50 — monitoring for 3 hours

If unmerged for >24 hours (compare Start Date or PR creation date):

• Post to Slack thread: ⏳ Build PR awaiting merge — <{pr_url}|PR> {owner_mention} (omit the trailing space + mention if Owner is empty)

If unmerged <24 hours: do nothing, check next run.

Phase 2: Validation (Status = 5-live:validating)

Calculate hours since Start Date. If < 3 hours, skip — check next run.

After 3 full hours, run health checks:

Run queries V1 and M1 from monitoring_queries.sql.

Check:

• Both variants receiving traffic (enrollment counts > 0)
• Split ratio approximately 50/50 (±5% tolerance) once the validation window has at least 100 enrolled visitors. For smaller samples, treat split imbalance as directional only; do not hold a low-volume experiment solely for a noisy split if both variants are receiving traffic.
• Variant URL returns 200
• Control has conversions

When using M1/M3 or custom adaptations of them, pre-aggregate analytics.website_conversion to one row per anonymous_id + pageview_date using boolean LOGICAL_OR fields before joining to assignments. Do not count raw joined conversion rows directly, because duplicate conversion rows can overstate conversions and trigger incorrect ship/kill decisions.

If healthy:

• Update Status → 6-live:running
• Post to thread: ✅ Validation passed after 3 hours — experiment running normally

If unhealthy:

• Post to thread with issue details: ⚠️ Validation issue on {experiment_id}: {description}. Holding at validating. {owner_mention} (omit the trailing space + mention if Owner is empty)
• Keep status at 5-live:validating — next run will re-check

Phase 3: Running (Status = 6-live:running)

3.1 — Query experiment data

Run monitoring queries M1 and M3 from monitoring_queries.sql. Adapt queries to use the Primary Metric from the Page Registry if it differs from the default did_download_same_day.

If adapting these queries, keep the same conversion de-duplication pattern: aggregate analytics.website_conversion by anonymous_id and pageview_date first, then join the aggregated conversion rows to the first assignment date. Count visitors from assignments and conversions from the aggregated boolean fields.

3.2 — Health checks

Read Method from the Notion entry to determine analysis type.

python3 .warp/skills/cro/experiment_monitor/scripts/experiment.py monitor \
  --control-n {{CONTROL_N}} \
  --control-conv {{CONTROL_CONVERSIONS}} \
  --variant-n {{VARIANT_N}} \
  --variant-conv {{VARIANT_CONVERSIONS}} \
  --target-n {{TARGET_N_PER_VARIANT}}

For Bayesian:

python3 .warp/skills/cro/experiment_monitor/scripts/experiment.py bayesian \
  --control-n {{CONTROL_N}} \
  --control-conv {{CONTROL_CONVERSIONS}} \
  --variant-n {{VARIANT_N}} \
  --variant-conv {{VARIANT_CONVERSIONS}} \
  --day {{CURRENT_DAY}} \
  --max-days {{MAX_DAYS}}

Only post to Slack if:

• A health issue is detected (split imbalance, zero conversions, extreme lift)
• The experiment has reached conclusion criteria (transitioning status)
• A severe regression guardrail is triggered (~50% drop sustained 3+ days)

Do NOT post routine status updates on every run.

3.3 — Check conclusion criteria

Frequentist: complete when both variants reach target sample size.

Bayesian: complete when decision is ship or kill (or max_days reached).

If concluded:

• Update Status → 7-analyzing
• This triggers the analyze agent via Notion automation → bud_server
• Do NOT post to Slack. The analyze agent’s 🟢 Experiment Complete message will cover the conclusion and result in a single post.

If not concluded: update Notion with current progress (enrollment counts) and continue. No Slack post.

Phase 4: Ship/Kill PR Merge Detection (Status = 7-analyzing)

The analyze skill creates the ship/kill PR and stores the URL in the Roadmap's Ship/Kill PR field.

Check if that PR has been merged:

gh pr view {{SHIP_KILL_PR_URL}} --repo your-org/your-marketing-site --json state,mergedAt

If merged:

• Read the Result field from the Notion entry.
• Verify the production deployment containing the merged ship/kill PR is live before touching production Sanity. If the deployment is not live yet, keep Status at 7-analyzing and check again next run. This prevents production code from routing to a variant page that has already been deleted.
• Apply the post-merge production Sanity mutations from the ship/kill PR body, scoped only to this experiment's artifacts:
  • ship: patch the production control page document to the promoted content from the merged markdown, then delete or mark completed the generated variant page document and matching active experiment document.
  • loss, no-ship, or inconclusive: leave the control page untouched, then delete or mark completed the generated variant page document and matching active experiment document.
• Verify production Sanity cleanup: no active experiment remains for {{EXPERIMENT_ID}}; for ship, the control page serves the promoted content; for no-ship/inconclusive, the control page remains unchanged.
• Update Status to the final value based on Result:
  • win or ship → 9-shipped
  • loss or no-ship → 8-no-ship
  • inconclusive → 8-inconclusive
• Set End Date → today.
• Do NOT post to Slack. The analyze agent’s 🟢 Experiment Complete message already reported the result. No additional notification needed unless production Sanity cleanup fails; if cleanup fails, post a warning in the thread with the blocker.

If no Ship/Kill PR URL yet: the analyze agent may still be running. Do nothing.

If PR exists but unmerged: check how long it's been. If >24 hours, nudge: ⏳ Ship/Kill PR awaiting merge — <{pr_url}|PR> {owner_mention} (omit the trailing space + mention if Owner is empty).

Important Notes

• No ramp phase. Experiments start at 50/50 when the build PR merges. Validation is a health check, not a traffic ramp.
• Ship/kill cleanup waits for deploy. Do not delete variant pages or disable experiment documents until the production deployment containing the ship/kill PR is live.
• Slack discipline. Only post when status transitions or something goes wrong. No per-run noise.
• Follow the Slack Thread TS protocol from config-slack.md for all posts.
• Slack token: only $EXPERIMENT_BUD_SLACK_BOT_TOKEN. See config-slack.md.
• CRITICAL: Never push directly to main on your-org/your-marketing-site.
• The backlog_monitor (first pass) handles chaining to the next experiment when a page becomes idle. This skill does not chain.
• Exit silently when no experiments need attention.