Sensitivity & Robustness Experimental Design Lens
Philosophical Mode: Robustness
Primary Question: "Which assumptions are load-bearing?"
Focus: Ablation Structure, Preprocessing Sensitivity, Metric Sensitivity, Hyperparameter Sensitivity, Distribution Shift
Arguments
/autoskillit:exp-lens-sensitivity-robustness [context_path] [experiment_plan_path]
- context_path (optional positional arg 1) — Absolute path to a lens context file
containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.
If provided, read this file before beginning analysis to obtain structured context.
If omitted, discover context by exploring the CWD.
- experiment_plan_path (optional positional arg 2) — Absolute path to the full
experiment plan. If provided, read for complete experimental methodology and design.
If omitted, locate the experiment plan by exploring the CWD.
When to Use
- Results may depend on specific preprocessing choices
- Need to verify robustness of conclusions across conditions
- Ablation study seems incomplete or cherry-picked
- User invokes
/autoskillit:exp-lens-sensitivity-robustness or /autoskillit:make-experiment-diag sensitivity
Critical Constraints
NEVER:
- Modify any source code or experiment files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Treat "untested" as equivalent to "robust"
- Create files outside
{{AUTOSKILLIT_TEMP}}/exp-lens-sensitivity-robustness/
- Run subagents in the background (
run_in_background: true is prohibited)
ALWAYS:
Build a full sensitivity matrix (choices x perturbation types)
Classify every analytic choice as load-bearing, minor, or untested
Flag cases where the most impactful choices are the least tested
Distinguish between ablations that were run and choices that were simply fixed
BEFORE creating any diagram, LOAD the /autoskillit:mermaid skill using the Skill tool - this is MANDATORY
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.
Write output to {{AUTOSKILLIT_TEMP}}/exp-lens-sensitivity-robustness/exp_diag_sensitivity_robustness_{YYYY-MM-DD_HHMMSS}.md
After writing the file, emit the structured output token as literal plain text with no
markdown formatting on the token name (the adjudicator performs a regex match):
diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-sensitivity-robustness/exp_diag_sensitivity_robustness_{...}.md
Analysis Workflow
Step 0: Parse optional arguments
If positional arg 1 (context_path) is provided and the file exists, read it to obtain
IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional
arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full
methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD
exploration for these fields if the context file supplies them.
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Analytic Choices Made
- Find all decision points in the analysis pipeline
- Look for: choice, option, default, parameter, threshold, method, alternative
Ablation Coverage
- Find which factors have been ablated
- Look for: ablation, without, remove, disable, vary, sweep, drop
Preprocessing Variations
- Find preprocessing steps that could be done differently
- Look for: normalize, tokenize, augment, crop, resize, filter, clean, impute
Hyperparameter Sensitivity
- Find which hyperparameters were tuned vs fixed
- Look for: learning_rate, batch_size, epochs, dropout, hidden_size, temperature, alpha, beta
Distribution/Environment Variations
- Find evidence of testing under different conditions
- Look for: shift, domain, transfer, cross, out_of_distribution, generalize, different
Step 2: Build the Sensitivity Matrix
Rows = analytic choices. Columns = perturbation types (remove, change, stress).
For each cell: Does the conclusion survive the perturbation?
Step 3: Classify Analytic Choices
CRITICAL — Analyze Assumption Load:
For every analytic choice:
- What happens if this choice were made differently?
- Is there evidence from ablations, sweeps, or prior literature?
- Are the most impactful choices the least tested?
Classify each choice as:
- Load-bearing: Conclusion changes if this choice changes
- Minor: Conclusion is robust to changes in this choice
- Untested: No evidence either way
Step 4: Create the Optional Perturbation Diagram
If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.
Direction: TB (choices flow down through perturbation to conclusion stability)
Subgraphs: "ANALYTIC CHOICES", "PERTURBATIONS TESTED", "CONCLUSION STABILITY"
Node Styling:
stateNode class: analytic choice nodes
handler class: perturbation type nodes
output class: stable conclusion nodes
gap class: load-bearing untested choice nodes
detector class: sensitivity threshold nodes
Step 5: Write Output
Write the analysis to: {{AUTOSKILLIT_TEMP}}/exp-lens-sensitivity-robustness/exp_diag_sensitivity_robustness_{YYYY-MM-DD_HHMMSS}.md (relative to the current working directory)
Output Template
# Sensitivity & Robustness Analysis: {Experiment Name}
**Lens:** Sensitivity & Robustness (Robustness)
**Question:** Which assumptions are load-bearing?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Sensitivity Matrix
| Analytic Choice | Remove | Change Value | Stress Test | Overall Classification |
|----------------|--------|-------------|-------------|------------------------|
| {choice name} | Stable / Fragile / Untested | Stable / Fragile / Untested | Stable / Fragile / Untested | Load-bearing / Minor / Untested |
## Load-Bearing Assumptions
| Assumption | Evidence Type | Impact if Changed | Tested? |
|------------|--------------|-------------------|---------|
| {assumption} | Ablation / Sweep / Literature / None | High / Medium / Low | Yes / No |
## Ablation Coverage Assessment
| Factor | Ablated? | Result | Interpretation |
|--------|----------|--------|----------------|
| {factor name} | Yes / No | {delta metric if yes} | Conclusion holds / Fragile / Unknown |
## Robustness Profile
| Dimension | Status | Notes |
|-----------|--------|-------|
| Preprocessing choices | Robust / Fragile / Untested | {detail} |
| Hyperparameter choices | Robust / Fragile / Untested | {detail} |
| Metric choices | Robust / Fragile / Untested | {detail} |
| Distribution shift | Robust / Fragile / Untested | {detail} |
## Perturbation Diagram (Optional)
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart TB
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Choices ["ANALYTIC CHOICES"]
CHOICE1["Preprocessing Choice<br/>━━━━━━━━━━<br/>{e.g., normalization}"]
CHOICE2["Model Choice<br/>━━━━━━━━━━<br/>{e.g., architecture}"]
LOAD_BEARING["Load-Bearing Choice<br/>━━━━━━━━━━<br/>{untested critical choice}"]
end
subgraph Perturbations ["PERTURBATIONS TESTED"]
PERTURB1["Remove<br/>━━━━━━━━━━<br/>Ablation"]
PERTURB2["Change Value<br/>━━━━━━━━━━<br/>Sensitivity sweep"]
THRESHOLD["Sensitivity Threshold<br/>━━━━━━━━━━<br/>{delta that changes conclusion}"]
end
subgraph Stability ["CONCLUSION STABILITY"]
STABLE["Stable Conclusion<br/>━━━━━━━━━━<br/>{holds under perturbation}"]
end
CHOICE1 --> PERTURB1
CHOICE2 --> PERTURB2
PERTURB1 --> THRESHOLD
PERTURB2 --> THRESHOLD
THRESHOLD --> STABLE
LOAD_BEARING -.->|"untested risk"| THRESHOLD
class CHOICE1,CHOICE2 stateNode;
class LOAD_BEARING gap;
class PERTURB1,PERTURB2 handler;
class THRESHOLD detector;
class STABLE output;
Color Legend:
| Color |
Category |
Description |
| Teal |
Analytic Choices |
Decision points in the pipeline |
| Yellow |
Load-Bearing Untested |
Critical choices with no perturbation evidence |
| Orange |
Perturbations |
Types of tests applied |
| Red |
Sensitivity Thresholds |
Points where conclusion may change |
| Dark Teal |
Stable Conclusions |
Results robust to perturbation |
Recommendations
- {Most urgent ablation to run — highest impact untested choice}
- {Preprocessing sensitivity test needed}
- {Distribution shift or domain generalization test needed}
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/autoskillit:make-experiment-diag` - Parent skill for lens selection
- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram
- `/autoskillit:exp-lens-estimand-clarity` - For clarifying which conclusions are being stress-tested
- `/autoskillit:exp-lens-iterative-learning` - For tracking robustness improvements across experiment iterations
1---2name: exp-lens-sensitivity-robustness3categories: [exp-lens]4activate_deps: [mermaid]5description: Create Sensitivity & Robustness experimental design analysis identifying load-bearing analytic choices and untested perturbations. Robustness lens answering "Which assumptions are load-bearing?"6hooks:7 PreToolUse:8 - matcher: "*"9 hooks:10 - type: command11 command: "echo 'Sensitivity & Robustness Lens - Analyzing assumption load...'"12 once: true13metadata:14 source: https://skillmd.com/skills/talont-org/exp-lens-sensitivity-robustness15---1617# Sensitivity & Robustness Experimental Design Lens1819**Philosophical Mode:** Robustness20**Primary Question:** "Which assumptions are load-bearing?"21**Focus:** Ablation Structure, Preprocessing Sensitivity, Metric Sensitivity, Hyperparameter Sensitivity, Distribution Shift2223## Arguments2425`/autoskillit:exp-lens-sensitivity-robustness [context_path] [experiment_plan_path]`2627- **context_path** (optional positional arg 1) — Absolute path to a lens context file28 containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.29 If provided, read this file before beginning analysis to obtain structured context.30 If omitted, discover context by exploring the CWD.31- **experiment_plan_path** (optional positional arg 2) — Absolute path to the full32 experiment plan. If provided, read for complete experimental methodology and design.33 If omitted, locate the experiment plan by exploring the CWD.3435## When to Use3637- Results may depend on specific preprocessing choices38- Need to verify robustness of conclusions across conditions39- Ablation study seems incomplete or cherry-picked40- User invokes `/autoskillit:exp-lens-sensitivity-robustness` or `/autoskillit:make-experiment-diag sensitivity`4142## Critical Constraints4344**NEVER:**45- Modify any source code or experiment files46- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves47- Treat "untested" as equivalent to "robust"48- Create files outside `{{AUTOSKILLIT_TEMP}}/exp-lens-sensitivity-robustness/`49- Run subagents in the background (`run_in_background: true` is prohibited)5051**ALWAYS:**52- Build a full sensitivity matrix (choices x perturbation types)53- Classify every analytic choice as load-bearing, minor, or untested54- Flag cases where the most impactful choices are the least tested55- Distinguish between ablations that were run and choices that were simply fixed56- BEFORE creating any diagram, LOAD the `/autoskillit:mermaid` skill using the Skill tool - this is MANDATORY57- If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.58- Write output to `{{AUTOSKILLIT_TEMP}}/exp-lens-sensitivity-robustness/exp_diag_sensitivity_robustness_{YYYY-MM-DD_HHMMSS}.md`59- After writing the file, emit the structured output token as **literal plain text** with no60 markdown formatting on the token name (the adjudicator performs a regex match):6162 ```63 diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-sensitivity-robustness/exp_diag_sensitivity_robustness_{...}.md64 ```6566---6768## Analysis Workflow6970### Step 0: Parse optional arguments7172If positional arg 1 (context_path) is provided and the file exists, read it to obtain73IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional74arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full75methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD76exploration for these fields if the context file supplies them.7778### Step 1: Launch Parallel Exploration Subagents7980Spawn Explore subagents to investigate:8182**Analytic Choices Made**83- Find all decision points in the analysis pipeline84- Look for: choice, option, default, parameter, threshold, method, alternative8586**Ablation Coverage**87- Find which factors have been ablated88- Look for: ablation, without, remove, disable, vary, sweep, drop8990**Preprocessing Variations**91- Find preprocessing steps that could be done differently92- Look for: normalize, tokenize, augment, crop, resize, filter, clean, impute9394**Hyperparameter Sensitivity**95- Find which hyperparameters were tuned vs fixed96- Look for: learning_rate, batch_size, epochs, dropout, hidden_size, temperature, alpha, beta9798**Distribution/Environment Variations**99- Find evidence of testing under different conditions100- Look for: shift, domain, transfer, cross, out_of_distribution, generalize, different101102### Step 2: Build the Sensitivity Matrix103104Rows = analytic choices. Columns = perturbation types (remove, change, stress).105106For each cell: Does the conclusion survive the perturbation?107108### Step 3: Classify Analytic Choices109110**CRITICAL — Analyze Assumption Load:**111For every analytic choice:112- What happens if this choice were made differently?113- Is there evidence from ablations, sweeps, or prior literature?114- Are the most impactful choices the least tested?115116Classify each choice as:117- **Load-bearing**: Conclusion changes if this choice changes118- **Minor**: Conclusion is robust to changes in this choice119- **Untested**: No evidence either way120121### Step 4: Create the Optional Perturbation Diagram122123If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.124125**Direction:** `TB` (choices flow down through perturbation to conclusion stability)126127**Subgraphs:** "ANALYTIC CHOICES", "PERTURBATIONS TESTED", "CONCLUSION STABILITY"128129**Node Styling:**130- `stateNode` class: analytic choice nodes131- `handler` class: perturbation type nodes132- `output` class: stable conclusion nodes133- `gap` class: load-bearing untested choice nodes134- `detector` class: sensitivity threshold nodes135136### Step 5: Write Output137138Write the analysis to: `{{AUTOSKILLIT_TEMP}}/exp-lens-sensitivity-robustness/exp_diag_sensitivity_robustness_{YYYY-MM-DD_HHMMSS}.md` (relative to the current working directory)139140---141142## Output Template143144```markdown145# Sensitivity & Robustness Analysis: {Experiment Name}146147**Lens:** Sensitivity & Robustness (Robustness)148**Question:** Which assumptions are load-bearing?149**Date:** {YYYY-MM-DD}150**Scope:** {What was analyzed}151152## Sensitivity Matrix153154| Analytic Choice | Remove | Change Value | Stress Test | Overall Classification |155|----------------|--------|-------------|-------------|------------------------|156| {choice name} | Stable / Fragile / Untested | Stable / Fragile / Untested | Stable / Fragile / Untested | Load-bearing / Minor / Untested |157158## Load-Bearing Assumptions159160| Assumption | Evidence Type | Impact if Changed | Tested? |161|------------|--------------|-------------------|---------|162| {assumption} | Ablation / Sweep / Literature / None | High / Medium / Low | Yes / No |163164## Ablation Coverage Assessment165166| Factor | Ablated? | Result | Interpretation |167|--------|----------|--------|----------------|168| {factor name} | Yes / No | {delta metric if yes} | Conclusion holds / Fragile / Unknown |169170## Robustness Profile171172| Dimension | Status | Notes |173|-----------|--------|-------|174| Preprocessing choices | Robust / Fragile / Untested | {detail} |175| Hyperparameter choices | Robust / Fragile / Untested | {detail} |176| Metric choices | Robust / Fragile / Untested | {detail} |177| Distribution shift | Robust / Fragile / Untested | {detail} |178179## Perturbation Diagram (Optional)180181```mermaid182%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%183flowchart TB184 %% CLASS DEFINITIONS %%185 classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;186 classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;187 classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;188 classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;189 classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;190 classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;191 classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;192 classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;193 classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;194195 subgraph Choices ["ANALYTIC CHOICES"]196 CHOICE1["Preprocessing Choice<br/>━━━━━━━━━━<br/>{e.g., normalization}"]197 CHOICE2["Model Choice<br/>━━━━━━━━━━<br/>{e.g., architecture}"]198 LOAD_BEARING["Load-Bearing Choice<br/>━━━━━━━━━━<br/>{untested critical choice}"]199 end200201 subgraph Perturbations ["PERTURBATIONS TESTED"]202 PERTURB1["Remove<br/>━━━━━━━━━━<br/>Ablation"]203 PERTURB2["Change Value<br/>━━━━━━━━━━<br/>Sensitivity sweep"]204 THRESHOLD["Sensitivity Threshold<br/>━━━━━━━━━━<br/>{delta that changes conclusion}"]205 end206207 subgraph Stability ["CONCLUSION STABILITY"]208 STABLE["Stable Conclusion<br/>━━━━━━━━━━<br/>{holds under perturbation}"]209 end210211 CHOICE1 --> PERTURB1212 CHOICE2 --> PERTURB2213 PERTURB1 --> THRESHOLD214 PERTURB2 --> THRESHOLD215 THRESHOLD --> STABLE216 LOAD_BEARING -.->|"untested risk"| THRESHOLD217218 class CHOICE1,CHOICE2 stateNode;219 class LOAD_BEARING gap;220 class PERTURB1,PERTURB2 handler;221 class THRESHOLD detector;222 class STABLE output;223```224225**Color Legend:**226| Color | Category | Description |227|-------|----------|-------------|228| Teal | Analytic Choices | Decision points in the pipeline |229| Yellow | Load-Bearing Untested | Critical choices with no perturbation evidence |230| Orange | Perturbations | Types of tests applied |231| Red | Sensitivity Thresholds | Points where conclusion may change |232| Dark Teal | Stable Conclusions | Results robust to perturbation |233234## Recommendations2352361. {Most urgent ablation to run — highest impact untested choice}2372. {Preprocessing sensitivity test needed}2383. {Distribution shift or domain generalization test needed}239```240241---242243## Pre-Diagram Checklist244245Before creating the diagram, verify:246247- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool248- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)249- [ ] Diagram will include a color legend table250251---252253## Related Skills254255- `/autoskillit:make-experiment-diag` - Parent skill for lens selection256- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram257- `/autoskillit:exp-lens-estimand-clarity` - For clarifying which conclusions are being stress-tested258- `/autoskillit:exp-lens-iterative-learning` - For tracking robustness improvements across experiment iterations