Measurement Design

Measurement Design

This skill handles CF-05: Measurement Discipline from the release-decision framework.

Its job is to ensure every hypothesis has exactly one primary metric, a small set of guardrails, and an event schema that can produce evidence for the decision.

When to Activate

• Hypothesis exists but no primary metric is defined
• User names multiple competing success metrics
• User mixes goals, proxies, and diagnostics together
• Instrumentation does not exist for the desired outcome
• Project primaryMetric field is empty or contains a list

On Entry — Read Current State

Before doing any work, read the project from the database using the project-sync skill's get-experiment command.

Check these fields:

• If hypothesis is empty → redirect to hypothesis-design
• If primaryMetric is already defined and instrumentation is confirmed → may not need this skill
• If guardrails already exist → build on existing rather than overwriting

Core Principle

One primary metric decides the outcome. Everything else is a guardrail or diagnostic.

If two metrics both "decide" success, the decision is not yet sharp enough. Return to hypothesis-design.

Decision Actions

Define the primary metric

Ask: "If this experiment runs for 2 weeks and you had to make a go/no-go decision with ONE number, what is that number?"

The answer is the primary metric. One only.

Then capture it as a structured object — NOT a paragraph. The five fields are:

If the user gives a vague metric name (e.g. "signup rate"), probe until the event key, metric type, and aggregation are concrete. Don't proceed with a half-defined metric.

Define guardrails (2–3 maximum)

Ask: "What other metrics would concern you if they degraded significantly, even if the primary metric improved?"

Common guardrails:

• Error rate / p99 latency for the candidate variant
• User satisfaction score or support ticket volume
• A downstream conversion step after the primary metric

Each guardrail has the same five fields as the primary metric, plus:

Design the event

For each metric, define:

• Event name (what action fires it)
• Required properties (user_key, session_id, relevant context)
• Where in the user journey it fires
• Whether it needs to be associated with a flag evaluation for experiment analysis

Verify instrumentation completeness

Check: can the current codebase emit this event? If not, instrumentation must be built before exposure begins.

Operating Rules

• Do not allow exposure to start without confirmed instrumentation
• One primary metric only — push back on lists
• Guardrails protect against harm, not success. They should not be optimized for.
• When multiple experiments share a flag or user pool, verify traffic allocation strategy before calculating sample size. Sequential experiments get full traffic; concurrent experiments with mutual exclusion get only their slice. An underpowered experiment due to traffic splitting is worse than waiting for a sequential slot. See reversible-exposure-control → multi-experiment-traffic.md for patterns.
• Hand off to reversible-exposure-control when instrumentation is complete
• Hand off to evidence-analysis once data collection is underway

Persist State

Use Skill("project-sync", ...) to sync state. All three writes are required, and instrumentation must be confirmed before writing:

PRIMARY='{"name":"Signup conversion","event":"signup_completed","metricType":"binary","metricAgg":"once","description":"Proportion of visitors that sign up."}'
GUARDRAILS='[{"name":"Checkout abandonment","event":"checkout_abandoned","metricType":"binary","metricAgg":"once","direction":"increase_bad","description":"must not rise"}]'

npx tsx $HOME/.claude/skills/project-sync/scripts/sync.ts update-state <experiment-id> \
  --primaryMetric "$PRIMARY" \
  --guardrails "$GUARDRAILS" \
  --lastAction "Metrics defined"
npx tsx $HOME/.claude/skills/project-sync/scripts/sync.ts set-stage <experiment-id> measuring
npx tsx $HOME/.claude/skills/project-sync/scripts/sync.ts add-activity <experiment-id> --type stage_update --title "Metrics defined"

primaryMetric must be a JSON object with {name, event, metricType, metricAgg, description?}. The web UI renders each field as its own column (NAME / EVENT / TYPE / AGG) — do NOT jam a paragraph into name. Rationale goes in description.

guardrails must be a JSON array of objects with the primary-metric shape plus direction (increase_bad or decrease_bad). One entry per guardrail, never a single string or newline-separated text.

sync.ts update-state validates both fields' JSON shape and enums; if validation fails it will print what's missing and exit non-zero.

Execution Procedure

def design_measurement(project_id, user_message):
    state = Skill("project-sync", f"get-experiment {project_id}")
    if state.hypothesis in ("", None):
        Skill("hypothesis-design", project_id)
        return
    # --- primary metric ---
    # ask: "if you had ONE number to make a go/no-go decision, what is it?"
    primary_metric = elicit_primary_metric(state, user_message)
    # --- guardrails (2–3 max) ---
    guardrails = elicit_guardrails(state)
    # --- event design ---
    events = design_events(primary_metric, guardrails)
    # --- instrumentation gate ---
    instrumentation_confirmed = confirm_instrumentation(events)
    if not instrumentation_confirmed:
        say("Instrumentation must be confirmed before exposure begins.")
        return  # do not advance stage until confirmed
    primary_json = json.dumps({
        "name": primary_metric.name,
        "event": primary_metric.event,
        "metricType": primary_metric.metric_type,       # binary | continuous
        "metricAgg":  primary_metric.metric_agg,        # once | count | sum | average
        "description": primary_metric.rationale,
    })
    guardrails_json = json.dumps([
        {
            "name":        g.name,
            "event":       g.event,
            "metricType":  g.metric_type,
            "metricAgg":   g.metric_agg,
            "direction":   g.direction,                  # increase_bad | decrease_bad
            "description": g.rationale,
        }
        for g in guardrails
    ])
    assert Skill("project-sync", f'update-state {project_id} --primaryMetric {shlex.quote(primary_json)} --guardrails {shlex.quote(guardrails_json)} --lastAction "Metrics defined"').ok
    assert Skill("project-sync", f"set-stage {project_id} measuring").ok
    assert Skill("project-sync", f'add-activity {project_id} --type stage_update --title "Metrics defined"').ok
    Skill("reversible-exposure-control", project_id)

Signal Inference

Reference Files

• references/event-schema-design.md — TrackPayload shape, event naming conventions, metric-to-event mapping, anti-patterns
• references/tool-featbit-sdk.md — FeatBit SDK track() usage, experiment event association, sendToExperiment