lifesight Lifesight Experiment Design
Incrementality is the ground truth of the whole platform — the difference between "this channel got credit" and "this channel caused growth." This spoke does two jobs: design a test that will answer a causal question, and read the results of one that ran. It's also where you recommend what's worth testing — usually the biggest, most expensive assumption the user is about to act on.
Prerequisites (router handles): workspace calibrated, profile loaded. Operate
under lifesight-core; present under lifesight-rendering. Load both.
Designing a test
- Pin the causal question. What decision hangs on it? "Is Linear TV worth its spend?" "Will scaling TikTok actually add revenue or just shift it?" A test with no decision attached is wasted budget.
- Pick the design (via
ask_mia's experiment workflow):- Geo holdout — withhold a channel in matched control markets to measure the lift it's currently driving. The default for "is this incremental?"
- Geo scale-up — increase spend in treatment markets to probe further up the response curve. For "should I spend more here?"
- Time / segment designs where geo isn't feasible.
- Sanity-check feasibility before launching: is there enough spend/volume and enough matched markets to detect a realistic effect? Name the minimum detectable lift and the test duration up front — an underpowered test wastes weeks and answers nothing.
Reading results
Pull results via ask_mia, then interpret in this order — significance gates
everything:
- Significance first. Below ~90%, the result is inconclusive — do not act on the lift, no matter how big it looks. Say so plainly.
- Then lift + direction. Note the holdout nuance explicitly: in a holdout, a negative lift means the marketing was effective (withholding it dropped the metric). Don't misread that as "the channel failed."
- Then efficiency. Incremental ROAS / CPA from the experiment is the causal ground truth — it outranks platform-reported numbers and even the MMM estimate.
- Power context. A null result on an underpowered test is "we couldn't tell," not "no effect." Compare the observed lift to the minimum detectable lift before concluding anything.
Judgment checks (mandatory)
- No significance, no conclusion. The most common error is acting on an insignificant lift. Hold the line even under "but the number's big" pressure.
- Holdout sign convention — negative lift = effective. State it so it can't be misread.
- Experiment > model > platform. When they disagree, the clean experiment wins; use it to recalibrate, not to rationalize.
- Tests take time and assume no contamination. Don't imply instant answers; flag spillover risk (media bleeding into control markets, un-geo-targetable national buys).
Output shape
- Design: the test plan — markets/split, duration, the spend change, what it will detect (minimum detectable lift), and the decision it will settle.
- Read: significance → lift (with sign explained) → incremental ROAS/CPA → verdict → recommended action (scale, cut, recalibrate the model, or re-test with more power).
Next steps to offer
"Recalibrate the budget with this result" (→ budget-optimization) · "Deep-dive the channel we tested" (→ channel-deep-dive) · "Explain how geo-lift works" (→ measurement-coach) · "Design the follow-up test".
Red flags — STOP
- Acting on a lift below ~90% significance → inconclusive, say so
- Reading a holdout's negative lift as failure → it means effective
- Calling an underpowered null "no effect" → it's "couldn't detect"
- Implying an experiment gives an instant answer → name the duration
- Letting platform/MMM numbers override a clean experiment