# Stress Test

> Pressure-test any analysis plan or investigation design for methodological flaws before execution — wrong baselines, survivorship bias, missing segments, uncontrolled confounds, absent kill criteria. Use on `/stress-test`, "review / gut-check / stress test my analysis plan", or before committing significant time to a plan from any source. Produces a 7-point PASS/WARNING/FAIL diagnostic with an A-F grade.

- Skill: `ai-analyst-lab/stress-test` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ai-analyst-lab/stress-test`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ai-analyst-lab/stress-test/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ai-analyst-lab (https://skillmd.com/u/ai-analyst-lab)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/ai-analyst-lab/stress-test

---


# Skill: Stress Test

**Type:** Standalone — reviews any analysis plan for methodological flaws

---

## Purpose

Takes an analysis plan, investigation design, or analytical approach and pressure-tests it for hidden flaws — wrong baselines, survivorship bias, missing segments, uncontrolled confounds, and absent kill criteria. Acts as a "senior data scientist code review" for your analytical thinking.

This skill is **standalone** — it works on any analysis plan, whether produced by `/analysis-design`, written by the user, or pulled from an existing document.

---

## When to Use

- Before committing a week to executing an analysis plan
- Before presenting an analysis design to stakeholders
- When you've written an analysis brief and want a gut-check
- When reviewing someone else's analytical approach
- After an analysis came back with unexpected results (was the design flawed?)

---

## Inputs

| Input | Required | Source | Description |
|-------|----------|--------|-------------|
| `{{PLAN}}` | Yes | User or file path | The analysis plan to review. Can be a file path, pasted text, or a description of the approach |
| `{{CONTEXT}}` | No | User | Business context — what decision this analysis will inform |
| `{{AUDIENCE}}` | No | User | Who will consume the results (affects what counts as "fatal" vs. "nice to have") |
| `{{DATA_DESCRIPTION}}` | No | User | Description of available data, if not obvious from the plan |

---

## The 7-Point Stress Test

**Check data availability first.** A methodologically perfect plan that needs data the dataset lacks is unexecutable, so this check precedes the seven checkpoints:
1. Read the active dataset schema (`.knowledge/datasets/{active}/schema.md`)
2. Check if the plan's required fields/tables/dimensions exist in the dataset
3. If ANY required data is missing → HALT, skip checkpoints 1-6, jump straight to checkpoint 7, assign FAIL verdict with BLOCKER status, provide F grade, and stop
4. If all required data exists → proceed with checkpoints 1-7 in order

**Common data blockers to check for:**
- Plan requires device/platform segmentation → check if device, platform, or user_agent fields exist
- Plan requires funnel analysis → check if event-level tracking exists (page views, clicks, sessions)
- Plan requires traffic source attribution → check if source, medium, campaign, or referrer fields exist
- Plan requires geographic segmentation → check if country, region, city, or geo fields exist
- Plan assumes conversion rate → check if both numerator (orders, signups) AND denominator (sessions, page views) data exist

---

After confirming data availability, review the plan against each checkpoint. For each one, assign a verdict: PASS, WARNING, or FAIL.

### 1. Hypothesis Clarity

**Check:** Is there a specific, testable hypothesis? Or is this a fishing expedition?

| Verdict | Criteria |
|---------|----------|
| PASS | Hypothesis is falsifiable with a clear claim about cause and effect |
| WARNING | Hypothesis exists but is vague ("engagement is down") or has multiple interpretations |
| FAIL | No hypothesis — plan is "explore the data and see what we find" |

**Fix if FAIL:** "State what you expect to find AND what would prove you wrong."

### 2. Baseline Validity

**Check:** Is the comparison group / baseline appropriate?

| Verdict | Criteria |
|---------|----------|
| PASS | Baseline accounts for seasonality, concurrent changes, and selection effects |
| WARNING | Baseline is reasonable but has known gaps (e.g., no YoY comparison available) |
| FAIL | Baseline is naive (e.g., "compare to last week" without checking for holidays, campaigns, or deployments) |

Common traps to check:
- Comparing to a period with a holiday, major campaign, or outage
- Before/after with no control group
- Comparing cohorts of different sizes or compositions
- Using "last month" when the metric is seasonal

**Fix if FAIL:** "Identify 2-3 alternative baselines and explain why each is valid or invalid."

### 3. Survivorship Bias

**Check:** Does the analysis only look at entities that "survived" to be measured?

| Verdict | Criteria |
|---------|----------|
| PASS | Analysis includes churned/dropped users, failed transactions, abandoned sessions |
| WARNING | Survivorship risk is acknowledged but not controlled for |
| FAIL | Analysis only includes "active" or "completed" entities without acknowledging what's excluded |

Common traps:
- "Average revenue per user" that only counts users who made a purchase
- "Onboarding completion rate" that only counts users who started onboarding
- "Feature satisfaction" survey that only reaches users who didn't churn
- Retention analysis that excludes users who churned before the measurement window

**Fix if FAIL:** "Explicitly define the denominator. Who is EXCLUDED from this analysis and why?"

### 4. Segment Coverage

**Check:** Will the analysis check for segment-level differences that could reverse the overall finding?

| Verdict | Criteria |
|---------|----------|
| PASS | Plan includes segment breakdowns on 3+ dimensions with explicit Simpson's paradox check |
| WARNING | Plan checks 1-2 segments but doesn't systematically test all categorical dimensions |
| FAIL | Plan only reports overall/aggregate numbers |

Key segments to always check:
- Device (mobile/desktop/tablet)
- Platform (iOS/Android/web)
- User tenure (new/returning/power user)
- Geography (if multi-market)
- Traffic source / acquisition channel
- Product category / feature area
- Customer tier / plan type

**Fix if FAIL:** "Add segment breakdowns for every categorical dimension with >2 values. Flag any reversal."

### 5. Confound Identification

**Check:** Does the plan identify and control for concurrent changes?

| Verdict | Criteria |
|---------|----------|
| PASS | Plan lists all known concurrent changes and has a control strategy for each |
| WARNING | Plan acknowledges confounds exist but doesn't have a strategy to isolate them |
| FAIL | Plan doesn't mention confounds or assumes the hypothesized cause is the only thing that changed |

Questions to surface confounds:
- What else shipped/launched/changed in the same time period?
- Were there any data pipeline or tracking changes?
- Did marketing spend or campaign strategy shift?
- Were there external events (holidays, competitor moves, news)?
- Did the user population change (growth spike, new channel, geography expansion)?

**Fix if FAIL:** "List every change that happened in the same time window. For each, explain how you'll separate its effect."

### 6. Kill Criteria

**Check:** Does the plan define what would make you REJECT the hypothesis?

| Verdict | Criteria |
|---------|----------|
| PASS | Explicit thresholds for acceptance AND rejection, plus practical significance criteria |
| WARNING | Has acceptance criteria but no rejection criteria (can only "confirm," never "disprove") |
| FAIL | No criteria — "we'll see what the data shows" |

What good criteria look like:
- "If the effect is less than 2pp, it's not practically significant regardless of p-value"
- "If the drop is uniform across all segments, the widget hypothesis is rejected"
- "If the confound analysis shows the loyalty program explains >60% of the variance, the widget is not the primary cause"

**Fix if FAIL:** "Define the minimum effect size that matters AND what finding would disprove your hypothesis."

### 7. Output Alignment

**Check:** Will the analysis output actually answer the stakeholder's question? **AND can the analysis be executed with available data?**

| Verdict | Criteria |
|---------|----------|
| PASS | Output format matches stakeholder need AND all required data exists |
| WARNING | Output will be useful but may need translation for the audience, OR data exists but may have quality/completeness issues |
| FAIL | Output is descriptive ("here's what happened") when stakeholder needs prescriptive ("here's what to do"), OR **BLOCKER: required data does not exist** (e.g., plan requires "product category" but dataset has no category field) |

Common misalignment:
- Stakeholder wants a recommendation, plan produces a description
- Stakeholder wants dollar impact, plan produces percentages
- Stakeholder wants a yes/no, plan produces "it depends"
- Stakeholder wants a 1-page brief, plan will produce a 20-page report
- **BLOCKER: Plan requires data fields that don't exist in the dataset**

**Fix if FAIL (data blocker):** "HALT — this analysis cannot be executed. The dataset does not contain [required field]. Either locate the field in another table, request a data extension, or pivot to a different dimension that exists (e.g., analyze by customer segment instead of product category)."

**Fix if FAIL (alignment):** "Restate the stakeholder's actual question and work backwards to what the output must contain."

---

## Output Format

**Start with a 1-sentence executive summary at the top** (before the separator line) stating the grade and any blockers.

```
STRESS TEST RESULTS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[1-sentence summary: e.g., "GRADE D (2 FAILs, 2 WARNINGs) — significant redesign needed before execution" or "GRADE F — BLOCKER: required data does not exist"]

OVERALL GRADE: [A/B/C/D/F]

| # | Check | Verdict | Issue |
|---|-------|---------|-------|
| 1 | Hypothesis Clarity | PASS/WARNING/FAIL | ... |
| 2 | Baseline Validity | PASS/WARNING/FAIL | ... |
| 3 | Survivorship Bias | PASS/WARNING/FAIL | ... |
| 4 | Segment Coverage | PASS/WARNING/FAIL | ... |
| 5 | Confound Identification | PASS/WARNING/FAIL | ... |
| 6 | Kill Criteria | PASS/WARNING/FAIL | ... |
| 7 | Output Alignment | PASS/WARNING/FAIL | ... |

CRITICAL ISSUES (must fix before executing):
  - ... [If data blocker: "BLOCKER: Cannot execute — required data does not exist. [Field name] not found in dataset."]

WARNINGS (address if time allows):
  - ...

RECOMMENDED FIXES:
  1. ... [state what the fix eliminates, not a time estimate]
  2. ...
  3. ...
```

### Grading Scale

| Grade | Criteria |
|-------|---------|
| A | 0 FAILs, 0-1 WARNINGs — execute with confidence |
| B | 0 FAILs, 2-3 WARNINGs — execute but address warnings |
| C | 1 FAIL, any WARNINGs — fix the failure before executing |
| D | 2 FAILs — significant redesign needed |
| F | 3+ FAILs — start over with the Analysis Design Brief |

---

## Output File

Saves to: `working/stress_test_{{DATE}}.md`

---

## Examples

### Review a plan you wrote
```
/stress-test "I'm going to compare this month's conversion rate to last month's, broken by device, to see if the new checkout flow helped."
```

### Review a file
```
/stress-test working/investigation_plan_2026-03-28.md
```

### With context
```
/stress-test working/analysis_brief.md context="Board meeting Friday, need to decide whether to invest $500K in onboarding redesign" audience="CEO and CFO"
```

---

## Integration

- Works independently of `/analysis-design` — use it on ANY plan
- Can be chained: `/analysis-design` → `/stress-test` → fix → re-test
- Complements the **question-framing skill** (which writes the Analysis Design Spec) by reviewing the plan after it's written
- Pairs with **Confound Scanner agent** — the scanner finds threats proactively during planning; `/stress-test` reviews an existing plan retroactively