Stress Test

Skill: Stress Test

Type: Standalone — reviews any analysis plan for methodological flaws

Purpose

Takes an analysis plan, investigation design, or analytical approach and pressure-tests it for hidden flaws — wrong baselines, survivorship bias, missing segments, uncontrolled confounds, and absent kill criteria. Acts as a "senior data scientist code review" for your analytical thinking.

This skill is standalone — it works on any analysis plan, whether produced by /analysis-design, written by the user, or pulled from an existing document.

When to Use

• Before committing a week to executing an analysis plan
• Before presenting an analysis design to stakeholders
• When you've written an analysis brief and want a gut-check
• When reviewing someone else's analytical approach
• After an analysis came back with unexpected results (was the design flawed?)

Inputs

The 7-Point Stress Test

Check data availability first. A methodologically perfect plan that needs data the dataset lacks is unexecutable, so this check precedes the seven checkpoints:

1. Read the active dataset schema (.knowledge/datasets/{active}/schema.md)
2. Check if the plan's required fields/tables/dimensions exist in the dataset
3. If ANY required data is missing → HALT, skip checkpoints 1-6, jump straight to checkpoint 7, assign FAIL verdict with BLOCKER status, provide F grade, and stop
4. If all required data exists → proceed with checkpoints 1-7 in order

Common data blockers to check for:

• Plan requires device/platform segmentation → check if device, platform, or user_agent fields exist
• Plan requires funnel analysis → check if event-level tracking exists (page views, clicks, sessions)
• Plan requires traffic source attribution → check if source, medium, campaign, or referrer fields exist
• Plan requires geographic segmentation → check if country, region, city, or geo fields exist
• Plan assumes conversion rate → check if both numerator (orders, signups) AND denominator (sessions, page views) data exist

After confirming data availability, review the plan against each checkpoint. For each one, assign a verdict: PASS, WARNING, or FAIL.

1. Hypothesis Clarity

Check: Is there a specific, testable hypothesis? Or is this a fishing expedition?

Fix if FAIL: "State what you expect to find AND what would prove you wrong."

2. Baseline Validity

Check: Is the comparison group / baseline appropriate?

Common traps to check:

• Comparing to a period with a holiday, major campaign, or outage
• Before/after with no control group
• Comparing cohorts of different sizes or compositions
• Using "last month" when the metric is seasonal

Fix if FAIL: "Identify 2-3 alternative baselines and explain why each is valid or invalid."

3. Survivorship Bias

Check: Does the analysis only look at entities that "survived" to be measured?

Common traps:

• "Average revenue per user" that only counts users who made a purchase
• "Onboarding completion rate" that only counts users who started onboarding
• "Feature satisfaction" survey that only reaches users who didn't churn
• Retention analysis that excludes users who churned before the measurement window

Fix if FAIL: "Explicitly define the denominator. Who is EXCLUDED from this analysis and why?"

4. Segment Coverage

Check: Will the analysis check for segment-level differences that could reverse the overall finding?

Key segments to always check:

• Device (mobile/desktop/tablet)
• Platform (iOS/Android/web)
• User tenure (new/returning/power user)
• Geography (if multi-market)
• Traffic source / acquisition channel
• Product category / feature area
• Customer tier / plan type

Fix if FAIL: "Add segment breakdowns for every categorical dimension with >2 values. Flag any reversal."

5. Confound Identification

Check: Does the plan identify and control for concurrent changes?

Questions to surface confounds:

• What else shipped/launched/changed in the same time period?
• Were there any data pipeline or tracking changes?
• Did marketing spend or campaign strategy shift?
• Were there external events (holidays, competitor moves, news)?
• Did the user population change (growth spike, new channel, geography expansion)?

Fix if FAIL: "List every change that happened in the same time window. For each, explain how you'll separate its effect."

6. Kill Criteria

Check: Does the plan define what would make you REJECT the hypothesis?

What good criteria look like:

• "If the effect is less than 2pp, it's not practically significant regardless of p-value"
• "If the drop is uniform across all segments, the widget hypothesis is rejected"
• "If the confound analysis shows the loyalty program explains >60% of the variance, the widget is not the primary cause"

Fix if FAIL: "Define the minimum effect size that matters AND what finding would disprove your hypothesis."

7. Output Alignment

Check: Will the analysis output actually answer the stakeholder's question? AND can the analysis be executed with available data?

Common misalignment:

• Stakeholder wants a recommendation, plan produces a description
• Stakeholder wants dollar impact, plan produces percentages
• Stakeholder wants a yes/no, plan produces "it depends"
• Stakeholder wants a 1-page brief, plan will produce a 20-page report
• BLOCKER: Plan requires data fields that don't exist in the dataset

Fix if FAIL (data blocker): "HALT — this analysis cannot be executed. The dataset does not contain [required field]. Either locate the field in another table, request a data extension, or pivot to a different dimension that exists (e.g., analyze by customer segment instead of product category)."

Fix if FAIL (alignment): "Restate the stakeholder's actual question and work backwards to what the output must contain."

Output Format

Start with a 1-sentence executive summary at the top (before the separator line) stating the grade and any blockers.

STRESS TEST RESULTS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[1-sentence summary: e.g., "GRADE D (2 FAILs, 2 WARNINGs) — significant redesign needed before execution" or "GRADE F — BLOCKER: required data does not exist"]

OVERALL GRADE: [A/B/C/D/F]

| # | Check | Verdict | Issue |
|---|-------|---------|-------|
| 1 | Hypothesis Clarity | PASS/WARNING/FAIL | ... |
| 2 | Baseline Validity | PASS/WARNING/FAIL | ... |
| 3 | Survivorship Bias | PASS/WARNING/FAIL | ... |
| 4 | Segment Coverage | PASS/WARNING/FAIL | ... |
| 5 | Confound Identification | PASS/WARNING/FAIL | ... |
| 6 | Kill Criteria | PASS/WARNING/FAIL | ... |
| 7 | Output Alignment | PASS/WARNING/FAIL | ... |

CRITICAL ISSUES (must fix before executing):
  - ... [If data blocker: "BLOCKER: Cannot execute — required data does not exist. [Field name] not found in dataset."]

WARNINGS (address if time allows):
  - ...

RECOMMENDED FIXES:
  1. ... [state what the fix eliminates, not a time estimate]
  2. ...
  3. ...

Grading Scale

Output File

Saves to: working/stress_test_{{DATE}}.md

Examples

Review a plan you wrote

/stress-test "I'm going to compare this month's conversion rate to last month's, broken by device, to see if the new checkout flow helped."

Review a file

/stress-test working/investigation_plan_2026-03-28.md

With context

/stress-test working/analysis_brief.md context="Board meeting Friday, need to decide whether to invest $500K in onboarding redesign" audience="CEO and CFO"

Integration

• Works independently of /analysis-design — use it on ANY plan
• Can be chained: /analysis-design → /stress-test → fix → re-test
• Complements the question-framing skill (which writes the Analysis Design Spec) by reviewing the plan after it's written
• Pairs with Confound Scanner agent — the scanner finds threats proactively during planning; /stress-test reviews an existing plan retroactively