Cro Analyze

CRO Experiment Analysis

Configuration

Read .warp/skills/cro/references/config.md for Notion IDs, data access, and status labels. Read .warp/skills/cro/references/config-slack.md for Slack posting config (token, channels, Thread TS protocol).

Skill-specific config:

• Stats library: .warp/skills/cro/experiment_monitor/scripts/experiment.py
• Monitoring queries: .warp/skills/cro/experiment_monitor/scripts/monitoring_queries.sql
• Priors: .warp/skills/cro/references/priors.yaml

Input

bud_server passes these fields from the Notion webhook:

NOTION_PAGE_ID: <hypothesis-page-id>
EXPERIMENT_ID: example-hero-copy
PAGE_PATH: /example-page

Procedure

Step 1: Fetch experiment context

Fetch the full Roadmap entry from Notion. Extract: Experiment Desc, Page, Method, Experiment ID, Build PR, Dash, Start Date, Slack Thread TS, Hypothesis, Change Type, Change Description.

Preflight required credentials before doing any analysis work. $NOTION_INTEGRATION_TOKEN must be present for Notion reads/writes, and either $GOOGLE_OAUTH_TOKEN or $METABASE_API_KEY must be present for experiment data. $EXPERIMENT_BUD_SLACK_BOT_TOKEN must be present before Step 6. If any required credential is missing, stop immediately and report that the triggering run is missing required Oz team secrets; do not attempt to work around missing credentials by spawning child agents, using alternate Slack tokens, or switching data sources.

SEO-only scope guard. Experiment Bud is CRO-only. Read references/config.md → Experiment Scope. If this entry is clearly SEO-only, do not create an analysis report, ship/kill PR, or Slack result. Log that the live SEO-only experiment needs human cleanup and exit. Do not silently mutate production artifacts for SEO-only tests.

Also fetch the Page Registry entry for the page to get Primary Metric, Secondary Metric(s), and Owner. Resolve {owner_mention} from Owner per references/config.md → Page Owners; if Owner is empty, omit the {owner_mention} line from the Slack post in Step 6.

Step 2: Run final analysis

Query experiment data using M1 and M3 from monitoring_queries.sql.

When adapting M1/M3 for the page's Primary Metric, preserve the conversion de-duplication pattern from monitoring_queries.sql: aggregate analytics.website_conversion to one row per anonymous_id + pageview_date with LOGICAL_OR metric fields before joining to experiment assignments. Do not count raw joined conversion rows directly, because duplicate conversion rows can materially overstate conversions and produce incorrect ship/kill decisions.

Run the appropriate analysis command based on the Method field from the Notion entry:

Frequentist:

python3 .warp/skills/cro/experiment_monitor/scripts/experiment.py significance \
  --control-n {{CONTROL_N}} \
  --control-conv {{CONTROL_CONVERSIONS}} \
  --variant-n {{VARIANT_N}} \
  --variant-conv {{VARIANT_CONVERSIONS}}

Bayesian:

python3 .warp/skills/cro/experiment_monitor/scripts/experiment.py bayesian \
  --control-n {{CONTROL_N}} \
  --control-conv {{CONTROL_CONVERSIONS}} \
  --variant-n {{VARIANT_N}} \
  --variant-conv {{VARIANT_CONVERSIONS}} \
  --day {{CURRENT_DAY}} \
  --max-days {{MAX_DAYS}}

The script returns a recommendation (frequentist) or decision (Bayesian) field with the ship/kill outcome. Use that directly — the thresholds and min sample guardrails are implemented in the script.

Also check the M3 paid vs organic breakout. Flag if they move in opposite directions.

Step 3: Create analysis report

Create a subpage under Reports (see config.md for Reports page ID).

Title: YYYY-MM-DD {{PAGE_PATH}} {{EXPERIMENT_ID}} Experiment Analysis

Report structure:

• Result summary: ship/no-ship/inconclusive with key numbers
• Statistical results: For frequentist: lift, CI, p-value, sample sizes. For Bayesian: P(variant > control), expected uplift, risk of loss, posterior parameters.
• Segment breakouts: Paid vs organic, OS/device if relevant
• Key learning: What did we learn? What should we test next?

Save the report URL.

Step 4: Write to Notion databases

4a. Update Roadmap entry:

• Result → win, loss, or inconclusive
• Lift → relative lift value
• End Date → today
• Analysis Link → report URL

Do not update Status in this skill. Leave the Roadmap entry at 7-analyzing after analysis and PR creation. experiment_monitor Phase 4 is the only step allowed to move an analyzed experiment to 9-shipped, 8-no-ship, or 8-inconclusive, and only after the ship/kill PR has merged, the production deployment containing it is live, and scoped production Sanity cleanup/promotion has completed. This prevents pages from becoming eligible for the next experiment while code-defined experiment cleanup is still unmerged.

4b. Create Analysis Log entry:

• Title: same as report title
• Status: completed
• Date: today
• Pages Analyzed: {{PAGE_PATH}}
• Full Report: report URL
• Key Takeaways: 1-2 sentence summary

4c. Create Knowledge Base insight:

• Insight: {{EXPERIMENT_ID}} — {result} ({relative_lift}% lift)
• Insight Source: experiment
• Tags: experiment-result
• Page: {{PAGE_PATH}}
• Analysis Link: report URL
• Summary: key learning from the experiment

Step 5: Create ship/kill PR

All code and markdown changes go in a single PR against your-org/your-marketing-site base branch main. Sanity is the runtime source of truth, but do not apply destructive production Sanity mutations before the ship/kill PR has merged and deployed; production may still be running the code-defined experiment until that deployment is live. Use the [ship-kill]/{{EXPERIMENT_ID}} branch convention. Do not run broad production pushes.

If ship (variant wins):

1. Remove the experiment: delete your-marketing-site/src/lib/ab/experiments/<experiment-id>.experiment.ts, then run pnpm run generate:experiments to regenerate experiments/index.ts. The middleware will stop intercepting traffic on next deploy.
2. Promote the variant content to control. Copy src/content/pages/<slug>-v<n>.md's frontmatter blocks over src/content/pages/<slug>.md (keep the control's title, slug, and seoNoindex — do NOT carry the variant's seoNoindex: true over to control). Then delete src/content/pages/<slug>-v<n>.md.
3. Prepare the post-merge production Sanity plan. The PR body must include the exact control slug, variant slug, experiment ID, and production Sanity actions for experiment_monitor Phase 4. Do not patch production Sanity yet. After the ship/kill PR merges and the production deployment containing it is live, experiment_monitor will patch the production control page to the promoted content, then delete or mark completed the variant document and active experiment document.
4. PR title: [Ship] {{EXPERIMENT_ID}} — variant wins.

If no-ship (control wins or inconclusive):

1. Remove the experiment: delete your-marketing-site/src/lib/ab/experiments/<experiment-id>.experiment.ts, then run pnpm run generate:experiments to regenerate experiments/index.ts.
2. Delete the variant markdown: rm src/content/pages/<slug>-v<n>.md.
3. Prepare the post-merge production Sanity cleanup plan. The PR body must include the exact variant slug and experiment ID for experiment_monitor Phase 4. Do not delete production Sanity artifacts yet. After the ship/kill PR merges and the production deployment containing it is live, experiment_monitor will delete or mark completed the active experiment document and delete the variant page document. Control content is untouched.
4. PR title: [No-Ship] {{EXPERIMENT_ID}} — reverting to control.

PR body: full analysis results, recommendation, link to Notion entry and report, list of files changed, plus a Post-merge production Sanity plan containing the exact control slug, variant slug, experiment ID, and expected cleanup/promotion actions for experiment_monitor.

Store the PR URL in the Roadmap entry's Ship/Kill PR field.

Step 6: Post to Slack thread

Follow the Slack Thread TS protocol from config-slack.md. Post the result to the experiment's thread.

Token: use ONLY $EXPERIMENT_BUD_SLACK_BOT_TOKEN. Use the canonical snippet from config-slack.md:

import json, os, sys, urllib.request

token = os.environ.get('EXPERIMENT_BUD_SLACK_BOT_TOKEN', '').strip()
if not token:
    print("FATAL: EXPERIMENT_BUD_SLACK_BOT_TOKEN is unset. Refusing to post.", file=sys.stderr)
    sys.exit(1)

payload = {'channel': 'SLACK_NOTIFICATIONS_CHANNEL_ID', 'text': message_text}
if thread_ts:
    payload['thread_ts'] = thread_ts

req = urllib.request.Request(
    'https://slack.com/api/chat.postMessage',
    data=json.dumps(payload).encode('utf-8'),
    headers={'Authorization': f'Bearer {token}', 'Content-Type': 'application/json; charset=utf-8'},
)
with urllib.request.urlopen(req) as resp:
    result = json.loads(resp.read().decode('utf-8'))
    if not result.get('ok'):
        print(f"Slack post failed: {result.get('error')}", file=sys.stderr)
        sys.exit(1)

Message templates:

Omit the final {owner_mention} line entirely if Owner is empty.

Frequentist format:

{emoji} Experiment Complete — {experiment_desc}
{page_path} · {duration} days · {total_visitors} visitors
•  Control: {control_cvr}% · Variant: {variant_cvr}%
•  {relative_lift}% lift · p={p_value}
{one_line_learning}
🔗 <{pr_url}|Ship/Kill PR> · <{report_url}|Report>
{owner_mention}

Bayesian format:

{emoji} Experiment Complete — {experiment_desc}
{page_path} · {duration} days · {total_visitors} visitors
•  Control: {control_cvr}% · Variant: {variant_cvr}%
•  P(variant > control): {prob}% · Expected uplift: {uplift}%
{one_line_learning}
🔗 <{pr_url}|Ship/Kill PR> · <{report_url}|Report>
{owner_mention}

Use 🟢 for ship, 🔴 for no-ship, 🟡 for inconclusive.

Important Notes

• CRITICAL: Never push directly to main on your-org/your-marketing-site. Ship/kill changes go through PRs.
• CRITICAL: Never set final Roadmap status from this skill. cro-analyze records the result, report, and ship/kill PR but must leave Status = 7-analyzing. Final status is owned by experiment_monitor after PR merge, production deploy, and scoped Sanity cleanup.
• Sanity is the runtime source of truth. Removing an experiment file from experiments/ and the markdown variant file is necessary but not sufficient — the production Sanity experiment doc and affected page docs must also reflect the new state, or users will see stale variants.
• Do not mutate production Sanity before the ship/kill deploy is live. The code-defined experiment can keep routing until production has deployed the removal PR. Early deletion of the variant page can break assigned traffic.
• Experiment Bud production-write exception. General agents must not write production Sanity. Experiment Bud may write production Sanity only for the current experiment's control page, generated variant page, and matching experiment document, and only in the launch or post-merge ship/kill phases described here.
• The experiment_monitor skill's Phase 4 detects when the ship/kill PR is merged and updates the final status.
• Always cite specific numbers in the report and Slack message. No vague claims.
• Print the full analysis to the agent session output as well.