Experimental Design Diagram Selection
Select the right experimental design lens for your analysis. Each lens asks one primary question about an experiment's design, surfacing assumptions and failure modes specific to that epistemic dimension.
When to Use
- Want to visualize or audit experimental design methodology
- Need to select the right lens for a specific experimental concern
- User says
/autoskillit:make-experiment-diag
- User wants to examine a specific aspect of an experiment
Critical Constraints
NEVER:
- Skip the selection step — always confirm which lens to use
- Modify any source code or experimental artifacts
- Combine multiple lenses in a single invocation
ALWAYS:
- Present the selection table to help the user choose
- Match user description to the most relevant lens
- Load the selected lens skill using the Skill tool
- Follow the loaded skill's instructions exclusively; the delegated skill handles completion
Core Lenses (always recommended)
| Your experiment involves... |
Use this lens |
Question it answers |
Philosophical Mode |
| Unclear or shifting claims |
Estimand Clarity |
What exactly is the claim? |
Evidential |
| Causal attribution without explicit assumptions |
Causal Assumptions |
What causal assumptions support this design? |
Causal-Structural |
| Baseline quality or fairness |
Comparator Construction |
Is the comparator fair and relevant? |
Counterfactual |
| Data preprocessing, splits, or feature pipelines |
Pipeline Integrity |
Could data handling create optimistic bias? |
Integrity |
| Nondeterminism, seed sensitivity, run-to-run noise |
Variance & Stability |
Is the signal larger than the noise? |
Stability |
| Asymmetric tuning, compute, or engineering effort |
Fair Comparison |
Are alternatives compared symmetrically? |
Fairness |
| Reproducing results from artifacts alone |
Reproducibility & Artifacts |
Could an independent party reproduce this? |
Transparency |
| Metric choice, proxy validity, score interpretation |
Measurement Validity |
Do measurements justify the interpretation? |
Psychometric |
| Robustness to preprocessing or modeling choices |
Sensitivity & Robustness |
Which assumptions are load-bearing? |
Robustness |
| Generalization claims beyond the test suite |
Benchmark Representativeness |
Does this generalize beyond the test bed? |
Generalizability |
Extended Lenses (enable by experiment class)
| Your experiment involves... |
Use this lens |
Question it answers |
Philosophical Mode |
| Shared resources, network effects, spillovers |
Unit & Interference |
What is the unit, and can treatments spill over? |
Causal-Structural |
| Power, multiplicity, sequential monitoring |
Error Budget |
Are error risks sized and controlled? |
Statistical |
| Theory claims needing adversarial stress |
Severity Testing |
Would this design have caught the error? |
Falsificationist |
| Assignment mechanism, blocking, stratification |
Randomization & Blocking |
Where does comparability come from? |
Design-Structural |
| Confounds, history effects, co-interventions |
Validity Threats |
What alternative explanations survive? |
Adversarial |
| Multi-step exploration, adaptive allocation |
Iterative Learning |
How does this maximize learning per cost? |
Decision-Theoretic |
| Mixing discovery and confirmation in one study |
Exploratory vs Confirmatory |
Is this discovery or test? |
Boundary |
| Deployment risk, fairness, stakeholder harm |
Governance & Risk |
What risks arise from acting on this result? |
Governance |
Workflow
Step 1: Prompt User
Ask the user:
What aspect of your experimental design would you like to examine?
Example prompts:
- "I want to check if my causal claims are well-supported"
- "I need to verify my data pipeline doesn't leak"
- "Are my baselines fair comparisons?"
- "Could someone reproduce my experiment?"
- "Is my benchmark representative enough?"
Step 2: Match to Lens
Based on the user's description, match to the most appropriate lens using the selection tables above. If ambiguous, present the top 2-3 candidates and ask the user to choose.
Step 3: Load the Selected Lens
Use the Skill tool to load the selected /autoskillit:exp-lens-{slug} skill.
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with the exp-lens analysis. Abort this step and inform the user that the lens skill is unavailable.
Step 4: Execute the Loaded Skill
The loaded lens skill takes over and runs its full analysis workflow.
Alias Table
| Alias |
Lens |
| estimand |
exp-lens-estimand-clarity |
| causal |
exp-lens-causal-assumptions |
| comparator |
exp-lens-comparator-construction |
| pipeline |
exp-lens-pipeline-integrity |
| variance |
exp-lens-variance-stability |
| fairness |
exp-lens-fair-comparison |
| reproducibility |
exp-lens-reproducibility-artifacts |
| measurement |
exp-lens-measurement-validity |
| sensitivity |
exp-lens-sensitivity-robustness |
| benchmark |
exp-lens-benchmark-representativeness |
| unit |
exp-lens-unit-interference |
| error |
exp-lens-error-budget |
| severity |
exp-lens-severity-testing |
| randomization |
exp-lens-randomization-blocking |
| validity |
exp-lens-validity-threats |
| iterative |
exp-lens-iterative-learning |
| exploratory |
exp-lens-exploratory-confirmatory |
| governance |
exp-lens-governance-risk |
Related Skills
/autoskillit:mermaid - Shared diagram styling (loaded by individual lenses)
/autoskillit:make-arch-diag - Software architecture lens counterpart
/autoskillit:verify-diag - Verify diagram accuracy against codebase
1---2name: make-experiment-diag3categories: [exp-lens]4activate_deps: [exp-lens]5description: Interactive selection of experimental design lens for visualizing experiment methodology. Routes to the appropriate exp-lens-* skill.6hooks:7 PreToolUse:8 - matcher: "*"9 hooks:10 - type: command11 command: "echo 'make-experiment-diag - Selecting experimental design lens...'"12 once: true13metadata:14 source: https://skillmd.com/skills/talont-org/make-experiment-diag15---1617# Experimental Design Diagram Selection1819Select the right experimental design lens for your analysis. Each lens asks one primary question about an experiment's design, surfacing assumptions and failure modes specific to that epistemic dimension.2021## When to Use2223- Want to visualize or audit experimental design methodology24- Need to select the right lens for a specific experimental concern25- User says `/autoskillit:make-experiment-diag`26- User wants to examine a specific aspect of an experiment2728## Critical Constraints2930**NEVER:**31- Skip the selection step — always confirm which lens to use32- Modify any source code or experimental artifacts33- Combine multiple lenses in a single invocation3435**ALWAYS:**36- Present the selection table to help the user choose37- Match user description to the most relevant lens38- Load the selected lens skill using the Skill tool39- Follow the loaded skill's instructions exclusively; the delegated skill handles completion4041---4243## Core Lenses (always recommended)4445| Your experiment involves... | Use this lens | Question it answers | Philosophical Mode |46|-----------------------------|---------------|---------------------|--------------------| 47| Unclear or shifting claims | Estimand Clarity | What exactly is the claim? | Evidential |48| Causal attribution without explicit assumptions | Causal Assumptions | What causal assumptions support this design? | Causal-Structural |49| Baseline quality or fairness | Comparator Construction | Is the comparator fair and relevant? | Counterfactual |50| Data preprocessing, splits, or feature pipelines | Pipeline Integrity | Could data handling create optimistic bias? | Integrity |51| Nondeterminism, seed sensitivity, run-to-run noise | Variance & Stability | Is the signal larger than the noise? | Stability |52| Asymmetric tuning, compute, or engineering effort | Fair Comparison | Are alternatives compared symmetrically? | Fairness |53| Reproducing results from artifacts alone | Reproducibility & Artifacts | Could an independent party reproduce this? | Transparency |54| Metric choice, proxy validity, score interpretation | Measurement Validity | Do measurements justify the interpretation? | Psychometric |55| Robustness to preprocessing or modeling choices | Sensitivity & Robustness | Which assumptions are load-bearing? | Robustness |56| Generalization claims beyond the test suite | Benchmark Representativeness | Does this generalize beyond the test bed? | Generalizability |5758## Extended Lenses (enable by experiment class)5960| Your experiment involves... | Use this lens | Question it answers | Philosophical Mode |61|-----------------------------|---------------|---------------------|--------------------| 62| Shared resources, network effects, spillovers | Unit & Interference | What is the unit, and can treatments spill over? | Causal-Structural |63| Power, multiplicity, sequential monitoring | Error Budget | Are error risks sized and controlled? | Statistical |64| Theory claims needing adversarial stress | Severity Testing | Would this design have caught the error? | Falsificationist |65| Assignment mechanism, blocking, stratification | Randomization & Blocking | Where does comparability come from? | Design-Structural |66| Confounds, history effects, co-interventions | Validity Threats | What alternative explanations survive? | Adversarial |67| Multi-step exploration, adaptive allocation | Iterative Learning | How does this maximize learning per cost? | Decision-Theoretic |68| Mixing discovery and confirmation in one study | Exploratory vs Confirmatory | Is this discovery or test? | Boundary |69| Deployment risk, fairness, stakeholder harm | Governance & Risk | What risks arise from acting on this result? | Governance |7071---7273## Workflow7475### Step 1: Prompt User7677Ask the user:7879> What aspect of your experimental design would you like to examine?8081Example prompts:82- "I want to check if my causal claims are well-supported"83- "I need to verify my data pipeline doesn't leak"84- "Are my baselines fair comparisons?"85- "Could someone reproduce my experiment?"86- "Is my benchmark representative enough?"8788### Step 2: Match to Lens8990Based on the user's description, match to the most appropriate lens using the selection tables above. If ambiguous, present the top 2-3 candidates and ask the user to choose.9192### Step 3: Load the Selected Lens9394Use the Skill tool to load the selected `/autoskillit:exp-lens-{slug}` skill.9596If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with the exp-lens analysis. Abort this step and inform the user that the lens skill is unavailable.9798### Step 4: Execute the Loaded Skill99100The loaded lens skill takes over and runs its full analysis workflow.101102---103104## Alias Table105106| Alias | Lens |107|-------|------|108| estimand | exp-lens-estimand-clarity |109| causal | exp-lens-causal-assumptions |110| comparator | exp-lens-comparator-construction |111| pipeline | exp-lens-pipeline-integrity |112| variance | exp-lens-variance-stability |113| fairness | exp-lens-fair-comparison |114| reproducibility | exp-lens-reproducibility-artifacts |115| measurement | exp-lens-measurement-validity |116| sensitivity | exp-lens-sensitivity-robustness |117| benchmark | exp-lens-benchmark-representativeness |118| unit | exp-lens-unit-interference |119| error | exp-lens-error-budget |120| severity | exp-lens-severity-testing |121| randomization | exp-lens-randomization-blocking |122| validity | exp-lens-validity-threats |123| iterative | exp-lens-iterative-learning |124| exploratory | exp-lens-exploratory-confirmatory |125| governance | exp-lens-governance-risk |126127---128129## Related Skills130131- `/autoskillit:mermaid` - Shared diagram styling (loaded by individual lenses)132- `/autoskillit:make-arch-diag` - Software architecture lens counterpart133- `/autoskillit:verify-diag` - Verify diagram accuracy against codebase