Exploratory-Confirmatory Experimental Design Lens
Philosophical Mode: Boundary
Primary Question: "Is this discovery or test, and are norms aligned?"
Focus: Pre-specification, Analytic Flexibility, HARKing Detection, Garden of Forking Paths, Transparent Reporting
Arguments
/autoskillit:exp-lens-exploratory-confirmatory [context_path] [experiment_plan_path]
- context_path (optional positional arg 1) — Absolute path to a lens context file
containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.
If provided, read this file before beginning analysis to obtain structured context.
If omitted, discover context by exploring the CWD.
- experiment_plan_path (optional positional arg 2) — Absolute path to the full
experiment plan. If provided, read for complete experimental methodology and design.
If omitted, locate the experiment plan by exploring the CWD.
When to Use
- Study mixes exploration and confirmation without clear boundaries
- Post-hoc hypotheses presented as pre-specified
- Many analyses run but only significant ones reported
- User invokes
/autoskillit:exp-lens-exploratory-confirmatory or /autoskillit:make-experiment-diag exploratory
Critical Constraints
NEVER:
- Modify any source code files
- Create files outside
{{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/
- Run subagents in the background (
run_in_background: true is prohibited)
ALWAYS:
Map the full analytic timeline — what was decided before vs. after seeing data
Count forking paths honestly — every analysis choice is a potential fork
Distinguish genuine exploration (hypothesis-generating) from HARKing (hypothesis-after-results)
Flag absent preregistration as a finding without assuming bad faith
BEFORE creating any diagram, LOAD the /autoskillit:mermaid skill using the Skill tool - this is MANDATORY
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.
Write output to {{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/exp_diag_exploratory_confirmatory_{YYYY-MM-DD_HHMMSS}.md
After writing the file, emit the structured output token as literal plain text with no
markdown formatting on the token name (the adjudicator performs a regex match):
diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/exp_diag_exploratory_confirmatory_{...}.md
Analysis Workflow
Step 0: Parse optional arguments
If positional arg 1 (context_path) is provided and the file exists, read it to obtain
IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional
arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full
methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD
exploration for these fields if the context file supplies them.
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Pre-specified Plans
- Find pre-registration documents, analysis plans, hypothesis files
- Look for: preregister, analysis_plan, hypothesis, registered, protocol, spec
Analytic Flexibility
- Find places where multiple analysis paths were possible
- Look for: alternatively, could_also, option, variant, subset, sensitivity, robustness
Selective Reporting Signals
- Find evidence of selective reporting or cherry-picking
- Look for: not_significant, excluded, not_shown, supplementary, additional, hidden
Post-Hoc Rationalization
- Find language suggesting post-hoc hypothesis generation
- Look for: we_noticed, interestingly, surprisingly, unexpectedly, upon_inspection
Exploration-Confirmation Separation
- Find explicit statements about exploratory vs. confirmatory intent
- Look for: exploratory, pilot, hypothesis_generating, confirmatory, pre_specified
Step 2: Map Analytic Timeline
What was decided before vs. after seeing data? Where is the exploration/confirmation boundary? Count forking paths.
Step 3: Analyze Boundary Integrity
For every reported result: Was the analysis plan fixed pre-outcome? How many alternatives could have been run? Does reporting distinguish exploratory from confirmatory? Assess survivorship bias.
Step 4: Create the Diagram (Optional)
Direction: LR (time flows left to right). Pre-data decisions → Data collection → Post-data decisions → Reporting
Step 5: Write Output
Write the output to: {{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/exp_diag_exploratory_confirmatory_{YYYY-MM-DD_HHMMSS}.md (relative to the current working directory)
Pre-Diagram Checklist
Before creating the diagram, verify:
Related Skills
/autoskillit:make-experiment-diag - Parent skill/autoskillit:mermaid - MUST BE LOADED before creating diagram/autoskillit:exp-lens-severity-testing/autoskillit:exp-lens-sensitivity-robustness
1---2name: exp-lens-exploratory-confirmatory3description: Assess whether analytic decisions were pre-specified or post-hoc and whether exploratory/confirmatory norms are aligned. Boundary lens answering "Is this discovery or test, and are norms aligned?"4---56# Exploratory-Confirmatory Experimental Design Lens78**Philosophical Mode:** Boundary9**Primary Question:** "Is this discovery or test, and are norms aligned?"10**Focus:** Pre-specification, Analytic Flexibility, HARKing Detection, Garden of Forking Paths, Transparent Reporting1112## Arguments1314`/autoskillit:exp-lens-exploratory-confirmatory [context_path] [experiment_plan_path]`1516- **context_path** (optional positional arg 1) — Absolute path to a lens context file17 containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.18 If provided, read this file before beginning analysis to obtain structured context.19 If omitted, discover context by exploring the CWD.20- **experiment_plan_path** (optional positional arg 2) — Absolute path to the full21 experiment plan. If provided, read for complete experimental methodology and design.22 If omitted, locate the experiment plan by exploring the CWD.2324## When to Use2526- Study mixes exploration and confirmation without clear boundaries27- Post-hoc hypotheses presented as pre-specified28- Many analyses run but only significant ones reported29- User invokes `/autoskillit:exp-lens-exploratory-confirmatory` or `/autoskillit:make-experiment-diag exploratory`3031## Critical Constraints3233**NEVER:**34- Modify any source code files35- Create files outside `{{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/`36- Run subagents in the background (`run_in_background: true` is prohibited)3738**ALWAYS:**39- Map the full analytic timeline — what was decided before vs. after seeing data40- Count forking paths honestly — every analysis choice is a potential fork41- Distinguish genuine exploration (hypothesis-generating) from HARKing (hypothesis-after-results)42- Flag absent preregistration as a finding without assuming bad faith43- BEFORE creating any diagram, LOAD the `/autoskillit:mermaid` skill using the Skill tool - this is MANDATORY44- If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.45- Write output to `{{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/exp_diag_exploratory_confirmatory_{YYYY-MM-DD_HHMMSS}.md`46- After writing the file, emit the structured output token as **literal plain text** with no47 markdown formatting on the token name (the adjudicator performs a regex match):4849 ```50 diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/exp_diag_exploratory_confirmatory_{...}.md51 ```5253---5455## Analysis Workflow5657### Step 0: Parse optional arguments5859If positional arg 1 (context_path) is provided and the file exists, read it to obtain60IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional61arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full62methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD63exploration for these fields if the context file supplies them.6465### Step 1: Launch Parallel Exploration Subagents6667Spawn Explore subagents to investigate:6869**Pre-specified Plans**70- Find pre-registration documents, analysis plans, hypothesis files71- Look for: preregister, analysis_plan, hypothesis, registered, protocol, spec7273**Analytic Flexibility**74- Find places where multiple analysis paths were possible75- Look for: alternatively, could_also, option, variant, subset, sensitivity, robustness7677**Selective Reporting Signals**78- Find evidence of selective reporting or cherry-picking79- Look for: not_significant, excluded, not_shown, supplementary, additional, hidden8081**Post-Hoc Rationalization**82- Find language suggesting post-hoc hypothesis generation83- Look for: we_noticed, interestingly, surprisingly, unexpectedly, upon_inspection8485**Exploration-Confirmation Separation**86- Find explicit statements about exploratory vs. confirmatory intent87- Look for: exploratory, pilot, hypothesis_generating, confirmatory, pre_specified8889### Step 2: Map Analytic Timeline9091What was decided before vs. after seeing data? Where is the exploration/confirmation boundary? Count forking paths.9293### Step 3: Analyze Boundary Integrity9495For every reported result: Was the analysis plan fixed pre-outcome? How many alternatives could have been run? Does reporting distinguish exploratory from confirmatory? Assess survivorship bias.9697### Step 4: Create the Diagram (Optional)9899**Direction:** LR (time flows left to right). Pre-data decisions → Data collection → Post-data decisions → Reporting100101### Step 5: Write Output102103Write the output to: `{{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/exp_diag_exploratory_confirmatory_{YYYY-MM-DD_HHMMSS}.md` (relative to the current working directory)104105---106107## Pre-Diagram Checklist108109Before creating the diagram, verify:110111- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool112- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)113- [ ] Diagram will include a color legend table114115---116117## Related Skills118119- `/autoskillit:make-experiment-diag` - Parent skill120- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram121- `/autoskillit:exp-lens-severity-testing`122- `/autoskillit:exp-lens-sensitivity-robustness`
TalonT-Org