Hypothesis Design

Hypothesis Design

This skill handles CF-02: Hypothesis Discipline from the release-decision framework.

Its job is to convert a goal into a testable, falsifiable statement before any implementation or measurement work begins.

When to Activate

• Goal exists but no causal claim links the change to the outcome
• User says "we think this will help" without explaining the mechanism
• hypothesis field is empty or non-falsifiable
• User is about to build without stating what they expect

On Entry — Read Current State

Use the project-sync skill's get-experiment command to load the current project state from the database. Check:

• goal and intent — were they set by intent-shaping? If empty, go back to intent-shaping first.
• hypothesis, change, variants, primaryMetric — are they already filled? If so, verify with the user whether to refine or start fresh.
• stage — confirms where the project is in the loop.

This read is required. Do not rely on conversation memory alone — the database is the canonical source.

Core Template

We believe [change X] will [move metric Y in direction Z] for [audience A], because [causal reason R].

Every component is required. A hypothesis without a causal reason is a hope, not a testable claim.

Validation Questions

Check each component:

1. Change X — Is this specific enough to implement? Could two engineers build the same thing from this description?
2. Metric Y — Is this measurable? Does instrumentation exist or can it be built?
3. Direction Z — Is the direction stated (increase / decrease / maintain)?
4. Audience A — Is the target audience specific enough to be segmented in analysis?
5. Reason R — Is the causal mechanism explicit? "Because users will like it" is not a reason.

Decision Actions

Draft the hypothesis

Work with the user to fill all five components. Ask about missing parts one at a time.

Test for falsifiability

Ask: "Under what conditions would we conclude this hypothesis was wrong?" If the answer is "none", it is not falsifiable.

Sharpen the metric claim

The hypothesis does not need a specific number at this stage. It needs a direction. Quantitative targets belong in the evaluation plan, not the hypothesis.

Operating Rules

• Do not proceed to implementation planning until all five components are present
• Do not conflate the hypothesis with the success threshold (that belongs in evidence-analysis)
• Hand off to reversible-exposure-control once hypothesis is confirmed

Persist State

Use Skill("project-sync", ...) to sync state to the web database. All three writes are required:

assert Skill("project-sync", f'update-state {experiment_id} --hypothesis "We believe [change] will [move metric] for [audience] because [reason]" --change "[specific change]" --variants "[control (annotation)|treatment (annotation)]" --primaryMetric "[metric name] — [rationale]" --lastAction "Hypothesis formed"').ok
assert Skill("project-sync", f"set-stage {experiment_id} hypothesis").ok
assert Skill("project-sync", f'add-activity {experiment_id} --type stage_update --title "Hypothesis formed"').ok

Execution Procedure

def design_hypothesis(project_id, user_message):
    state = Skill("project-sync", f"get-experiment {project_id}")
    if state.goal in ("", None):
        Skill("intent-shaping", project_id)
        return
    template = read("references/hypothesis-template.md")
    # fill all 5 components: change, metric, direction, audience, causal reason
    # ask about missing parts one at a time
    # falsifiability check: "under what conditions would we conclude this was wrong?"
    hypothesis = build_hypothesis(state.goal, template, user_message)
    assert Skill("project-sync", f'update-state {project_id} --hypothesis "{hypothesis.text}" --change "{hypothesis.change}" --variants "{hypothesis.control} (control)|{hypothesis.treatment} (treatment)" --primaryMetric "{hypothesis.metric} — {hypothesis.metric_rationale}" --lastAction "Hypothesis formed"').ok
    assert Skill("project-sync", f"set-stage {project_id} hypothesis").ok
    assert Skill("project-sync", f'add-activity {project_id} --type stage_update --title "Hypothesis formed"').ok
    Skill("reversible-exposure-control", project_id)

Signal Inference

Reference Files

• references/hypothesis-template.md — full template, good/bad examples, falsifiability check, what belongs here vs. in evidence-analysis