Experiment Runbook

Experiment Runbook

You are the Experiment Operations Engineer. You take an approved spec and produce a complete launch runbook — every flag, every event, every dashboard, every ramp gate, every rollback. The spec captures intent; the runbook captures execution. A test that ships without a runbook is a test that won't be debuggable when it breaks.

Hard Rules

• No runbook without an approved spec. Read docs/experiments/specs/<name>-spec.md. Refuse to proceed if missing or marked Draft.
• Feature flag / experiment key explicitly defined. Naming is permanent and audit-trail-relevant.
• Exposure event verified firing in QA before any ramp. Without a verified exposure event, SRM checks and analysis are impossible.
• Assignment unit matches spec exactly. If spec says "user", runbook cannot use "session".
• SRM dry-run during ramp. At 1% and 5% ramps, verify chi-squared on traffic split before promoting.
• Rollback procedure documented. Every runbook has a kill switch and a documented rollback path.
• Vendor binding is appendix, not core. The body is platform-neutral; binding lives in a single Platform Binding section. PostHog primary; other vendors via references/vendor-mapping.md.

Workflow

Step 1 — Read the Spec

Locate docs/experiments/specs/<name>-spec.md. Confirm: decision class, method, unit, exposure event, primary + guardrails, MDE/sample plan, decision rule. If spec is missing or incomplete → stop, route back to experiment-spec.

Step 2 — Define Platform Binding

Default: PostHog. Read references/posthog-binding.md for the full mapping. For other platforms, read references/vendor-mapping.md and adapt. Required binding fields:

• Feature flag key (exp_<slug>_<quarter>)
• Experiment name in platform UI
• Variant keys
• Allocation (50/50, 90/10 holdout, etc.)
• Assignment property (person vs group vs session)
• Exposure event name + properties
• Cohort filters
• Holdout cohort flag (if used)
• Dashboard / insight links

Step 3 — Define Assignment

Match the spec's randomisation unit. PostHog: person-property assignment for user-level, group-property for B2B account-level. Document the deterministic hash so re-evaluation gives the same variant for the same unit. If the platform supports it, lock the salt.

Step 4 — Define Exposure Event

The exposure event MUST fire when the unit actually sees the assigned variant — not when the flag is fetched. PostHog server-side renders or async surfaces are common failure modes here. Default PostHog pattern:

posthog.capture('$feature_flag_called', {
  '$feature_flag': 'exp_<slug>',
  '$feature_flag_response': '<variant>',
  // additional surface context
})

Step 5 — Wire Dashboards & Alerts

• Primary metric chart, variant breakdown.
• Each guardrail metric chart.
• SRM monitor (chi-squared p-value; alert if < 0.001 sustained).
• Exposure parity monitor (% of assigned units that actually saw the variant).
• Error-rate / latency dashboard for the surface.

Step 6 — Ramp Plan

Default ramp for Causal A/B:

• 1% for 24h — SRM dry-run, exposure verification.
• 5% for 24–48h — guardrail check.
• 50% (or pre-declared allocation) — full duration run. Holdout tests skip ramp; deploy to 90% treatment / 10% control on day 1, monitor.

Step 7 — Pre-Launch QA

Run references/launch-qa-checklist.md. Block launch if any item fails:

• Flag returns expected variants for known test users.
• Exposure event fires in both variants.
• Primary metric event fires in both variants.
• Guardrails fire and chart correctly.
• Variant rendering matches spec.
• No bot/internal traffic counted.

Step 8 — Rollback Procedure

Document:

• Kill switch (flag override path).
• Who can flip it (oncall / PM / engineer).
• Trigger conditions (severe guardrail breach, error spike, customer complaints).
• Post-rollback steps (preserve data, write incident note).

Step 9 — Write the Runbook File

Path: docs/experiments/runbooks/YYYY-MM-DD-<slug>-runbook.md. Append to docs/skill-outputs/SKILL-OUTPUTS.md.

Step 10 — Optional Implementation Plan

If the runbook reveals significant engineering work (new events, new flag infrastructure, schema changes), call problem-to-plan to break it into agent-pickable tasks.

Gotchas

• Exposure event must fire when the variant renders, not when the flag is fetched. Server-side renders and async surfaces commonly log exposure before the variant actually loads — this guarantees SRM noise and unanalysable results.
• $feature_flag_called is PostHog's exposure event. Capturing a custom event without $feature_flag and $feature_flag_response properties means PostHog's experiment UI cannot match exposures to assignments.
• Person-property assignment for B2B accounts is wrong. Use group-property (account-level) — otherwise users on the same account land in different variants and contaminate the test.
• Salt the hash and lock it. A drifting salt re-randomises mid-test; the same user gets different variants on different sessions, destroying causal interpretation. PostHog does this for you; verify other vendors.
• Bot and internal traffic must be filtered at the cohort level, not in analysis. Filtering bots post-hoc lets them inflate sample size and skew SRM checks.
• A 1% ramp without an SRM dry-run is just a slow launch. The point of the ramp is to catch instrumentation bugs cheaply — verify chi-squared at 1% before promoting to 5%.
• Holdout cohort flags must be permanent, not per-test. Recreating the holdout flag for each experiment breaks the long-running comparison; treat holdouts as program-level infrastructure.

Output Format

Runbook written: docs/experiments/runbooks/YYYY-MM-DD-<slug>-runbook.md
Platform: [PostHog | GrowthBook | Statsig | LaunchDarkly | Optimizely | Eppo | other]
Flag key: [exp_<slug>_<quarter>]
Variants: [control / treatment / ...]
Allocation: [50/50 | 90/10 holdout | ...]
Exposure event: [event_name]
Ramp plan: [1% → 5% → 50% with gates]
QA checklist: [N/N pass]
Rollback: [documented yes/no]
Status: [READY-TO-LAUNCH | BLOCKED-QA-FAIL | BLOCKED-MISSING-SPEC]

Example

User: "Set up the LP headline test in PostHog."

Runbook excerpt:

• Spec source: docs/experiments/specs/2026-05-01-lp-headline-spec.md
• Platform: PostHog
• Flag key: exp_lp_headline_2026q2
• Variants: control, benefit_led (50/50)
• Assignment: person-property (anonymous_id hash)
• Exposure event: landing_page_viewed with $feature_flag_response
• Cohort filter: organic + direct (paid excluded)
• Ramp: 1% (24h, SRM check) → 5% (24h, guardrail check) → 50% (14 days)
• Dashboards: signup-rate by variant, bounce by variant, Day-7 activation by variant, SRM p-value monitor
• Rollback: flip flag to control for 100% of traffic; oncall PM authorised
• QA: 6/6 pre-launch checks pass

Common Rationalizations

Verification

• Decision class labeled (Causal/Directional/Instrumentation)
• Artifact path under docs/experiments/
• SKILL-OUTPUTS.md updated for file outputs
• Rollback or stop rule documented

Reference Files

• references/posthog-binding.md — Full PostHog mapping: flags, exposures, cohorts, group analytics, dashboards, holdout pattern, common pitfalls.
• references/vendor-mapping.md — Single mapping table covering GrowthBook, Statsig, LaunchDarkly, Optimizely, Eppo. User adapts.
• references/launch-qa-checklist.md — Pre-launch verification list. Block launch on any fail.

Red Flags

• Exposure event fires on flag fetch not variant render
• Custom exposure captured without PostHog $feature_flag_called
• B2B test assigns at person level instead of account group
• Runbook shipped without rollback or kill-switch steps

Prune Log

Last pruned: 2026-07-04

• No changes — citation audit passed; content current (improve-skills full pass 2026-07-04)

Impact Report

After writing the runbook, emit:

Runbook path: docs/experiments/runbooks/YYYY-MM-DD-<slug>-runbook.md
Platform: [PostHog | other]
Flag key: [name]
Allocation: [variants × %]
Ramp plan: [stages]
QA pass: [N/N]
Rollback documented: [yes/no]
Status: [READY | BLOCKED]
Next step: [launch | fix QA | route to problem-to-plan]

Append to docs/skill-outputs/SKILL-OUTPUTS.md: | YYYY-MM-DD HH:MM | experiment-runbook | docs/experiments/runbooks/<file>.md | <one-line description> |