Agenthog Experiment

Run an experiment with AgentHog

An experiment is an analysis window on a multivariate feature flag. The flag delivers variants (deterministic per-user hashing in the SDK — no assignment service); the experiment compares treatments against a control on a metric and reports each arm's chance to beat it. Everything runs through the ah CLI; every mutation needs a write-scope token.

Follow the phases in order. Each ends with a check — do not proceed past a failed check.

0. Preconditions

• The AgentHog SDK is integrated and verified in the target app (events arrive in ah events). If not: stop and run the agenthog-integrate skill first.
• ah is authenticated with write scope: ah whoami shows the project and scope: write. If read-only, ask the user to run ah login --write.
• You know the decision metric: one event name that already exists (check ah events top) or a goal (ah goals list). If the metric event doesn't exist yet, instrument it and verify it arrives before doing anything else.

1. Create the flag

ah flags create checkout_cta --variants control:50,b:50 --desc "new checkout CTA copy"

• Key: lowercase [a-z0-9_-], ≤64 chars. Variant names the same. One arm should be named control unless the user says otherwise.
• A/B/C: --variants control:34,b:33,c:33. Percentage rollout of a non-experiment feature: ah flags create new_nav (boolean) then ah flags rollout new_nav 10.
• Start experiments at full traffic (--traffic 100) unless the user wants a limited blast radius; you can ramp up later — ramping up never reshuffles assignments.

2. Wire the read into the app — with a hard fallback

The pattern is identical on every platform: read the flag, branch, and make the fallback path the control experience. undefined means "no ruleset yet / not enrolled / killed" — the app must work when the flag system says nothing.

Web (window.agenthog), React Native (useAgentHog()), Capacitor (AgentHog):

await agenthog.flagsReady()                 // avoids the first-visit undefined window
const v = agenthog.flag('checkout_cta')     // 'control' | 'b' | undefined
if (v === 'b') { /* treatment */ } else { /* control AND fallback */ }

Never emit $exposure events or $ff/* props yourself — the SDK does both on the first flag() read. Never branch on anything except the flag() return value.

Check before continuing (exposure must flow BEFORE the window opens):

ah events --name '$exposure' --since 1h            # rows appear as you exercise the app
ah events --by flag:checkout_cta --since 1h        # events split by variant

Use agenthog.overrideFlag('checkout_cta', 'b') to force each arm during this check — overrides never pollute experiment data. Clear with overrideFlag(key, null).

3. Open the experiment window

ah experiments start checkout_cta --metric checkout_completed \
  [--min-days 1|7] [--secondary e1,e2] [--guardrail app_error:up] [--hypothesis "..."]

• --min-days gates only the verdict badge. Choose by metric horizon: 1 when the metric fires in the same session as exposure (onboarding, activation, checkout); keep the default 7 for anything with weekly rhythm (retention, repeat usage).
• Guardrail direction is the REGRESSION direction: errors:up, activation:down.
• The command prints a 30-day baseline and required-users-per-arm for 5/10/20% MDEs. Relay this to the user — if the estimate says months, say so now, not after.

4. Read results honestly

ah experiments results checkout_cta [--json]

• collecting (n of 100; day d of N) — report progress, do NOT characterize which arm is ahead. Small-sample rates mislead.
• readable, no ★ — report chance-to-beat and credible intervals with the caveats printed. "68% likely better" is not a winner; say what would settle it (more users or more days).
• ★ likely winner — safe to recommend shipping. Quote the lift and chance-to-beat.
• Repeat any caveat: lines verbatim (day-of-week coverage, excluded multi-variant users). If a guardrail shows ⚠, lead with that — a primary-metric win that spikes errors is not a win.
• --json field names are a stable contract for scripting.

Do not stop a live experiment, change its weights, or declare winners the tool did not declare, unless the user explicitly asks (weight edits require --force for exactly this reason).

5. Ship, iterate, or abandon

• Ship: ah experiments stop checkout_cta --winner b --conclusion "..." then ah flags weights checkout_cta b:100. The flag now serves the winner to everyone.
• Iterate: stop (no winner), adjust variants/copy, ah experiments start again — a fresh window on the same flag; results never mix across windows.
• Abandon / emergency: ah flags disable checkout_cta is the kill switch — every client falls back to code defaults within ~2 minutes.

6. Retire

Weeks later, once the winner is the only path anyone gets:

1. Delete the losing branch and the flag() read from the code; the winner's code becomes the only path.
2. ah flags archive checkout_cta — leaves the SDK payload, keeps history.
3. Confirm: ah events --name '$exposure' --since 48h shows nothing for the key after the archived deploy is live.

You are done when

• ah flags show <key> matches what the user asked for (variants, traffic, state)
• $exposure events arrive and ah events --by flag:<key> splits by variant
• The experiment window is live with the right metric and --min-days, and the user knows the expected duration from the baseline hint
• You reported results with their gates and caveats intact — no early winners
• On ship: winner at 100% weight, experiment stopped with --winner, and (when retiring) the flag archived and its code path deleted

Timeline of everything you did: ah changes list --kind experiment. Flags and results are also visible at https://agenthog.io/flags. To write the results up as a shared report the whole organization reads in the dashboard, use the agenthog-report skill, which ships alongside this one.