Exploratory-Confirmatory Experimental Design Lens
Philosophical Mode: Boundary
Primary Question: "Is this discovery or test, and are norms aligned?"
Focus: Pre-specification, Analytic Flexibility, HARKing Detection, Garden of Forking Paths, Transparent Reporting
Arguments
/autoskillit:exp-lens-exploratory-confirmatory [context_path] [experiment_plan_path]
- context_path (optional positional arg 1) — Absolute path to a lens context file
containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.
If provided, read this file before beginning analysis to obtain structured context.
If omitted, discover context by exploring the CWD.
- experiment_plan_path (optional positional arg 2) — Absolute path to the full
experiment plan. If provided, read for complete experimental methodology and design.
If omitted, locate the experiment plan by exploring the CWD.
When to Use
- Study mixes exploration and confirmation without clear boundaries
- Post-hoc hypotheses presented as pre-specified
- Many analyses run but only significant ones reported
- User invokes
/autoskillit:exp-lens-exploratory-confirmatory or /autoskillit:make-experiment-diag exploratory
Critical Constraints
NEVER:
- Modify any source code files
- Create files outside
{{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/
- Run subagents in the background (
run_in_background: true is prohibited)
ALWAYS:
Map the full analytic timeline — what was decided before vs. after seeing data
Count forking paths honestly — every analysis choice is a potential fork
Distinguish genuine exploration (hypothesis-generating) from HARKing (hypothesis-after-results)
Flag absent preregistration as a finding without assuming bad faith
BEFORE creating any diagram, LOAD the /autoskillit:mermaid skill using the Skill tool - this is MANDATORY
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.
Write output to {{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/exp_diag_exploratory_confirmatory_{YYYY-MM-DD_HHMMSS}.md
After writing the file, emit the structured output token as literal plain text with no
markdown formatting on the token name (the adjudicator performs a regex match):
diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/exp_diag_exploratory_confirmatory_{...}.md
Analysis Workflow
Step 0: Parse optional arguments
If positional arg 1 (context_path) is provided and the file exists, read it to obtain
IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional
arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full
methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD
exploration for these fields if the context file supplies them.
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Pre-specified Plans
- Find pre-registration documents, analysis plans, hypothesis files
- Look for: preregister, analysis_plan, hypothesis, registered, protocol, spec
Analytic Flexibility
- Find places where multiple analysis paths were possible
- Look for: alternatively, could_also, option, variant, subset, sensitivity, robustness
Selective Reporting Signals
- Find evidence of selective reporting or cherry-picking
- Look for: not_significant, excluded, not_shown, supplementary, additional, hidden
Post-Hoc Rationalization
- Find language suggesting post-hoc hypothesis generation
- Look for: we_noticed, interestingly, surprisingly, unexpectedly, upon_inspection
Exploration-Confirmation Separation
- Find explicit statements about exploratory vs. confirmatory intent
- Look for: exploratory, pilot, hypothesis_generating, confirmatory, pre_specified
Step 2: Map Analytic Timeline
What was decided before vs. after seeing data? Where is the exploration/confirmation boundary? Count forking paths.
Step 3: Analyze Boundary Integrity
For every reported result: Was the analysis plan fixed pre-outcome? How many alternatives could have been run? Does reporting distinguish exploratory from confirmatory? Assess survivorship bias.
Step 4: Create the Diagram (Optional)
Direction: LR (time flows left to right). Pre-data decisions → Data collection → Post-data decisions → Reporting
Step 5: Write Output
Write the output to: {{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/exp_diag_exploratory_confirmatory_{YYYY-MM-DD_HHMMSS}.md (relative to the current working directory)
Pre-Diagram Checklist
Before creating the diagram, verify:
Related Skills
/autoskillit:make-experiment-diag - Parent skill
/autoskillit:mermaid - MUST BE LOADED before creating diagram
/autoskillit:exp-lens-severity-testing
/autoskillit:exp-lens-sensitivity-robustness
1---2name: exp-lens-exploratory-confirmatory3categories: [exp-lens]4activate_deps: [mermaid]5description: Assess whether analytic decisions were pre-specified or post-hoc and whether exploratory/confirmatory norms are aligned. Boundary lens answering "Is this discovery or test, and are norms aligned?"6hooks:7 PreToolUse:8 - matcher: "*"9 hooks:10 - type: command11 command: "echo 'Exploratory-Confirmatory Lens - Analyzing boundary integrity...'"12 once: true13metadata:14 source: https://skillmd.com/skills/talont-org/exp-lens-exploratory-confirmatory15---1617# Exploratory-Confirmatory Experimental Design Lens1819**Philosophical Mode:** Boundary20**Primary Question:** "Is this discovery or test, and are norms aligned?"21**Focus:** Pre-specification, Analytic Flexibility, HARKing Detection, Garden of Forking Paths, Transparent Reporting2223## Arguments2425`/autoskillit:exp-lens-exploratory-confirmatory [context_path] [experiment_plan_path]`2627- **context_path** (optional positional arg 1) — Absolute path to a lens context file28 containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.29 If provided, read this file before beginning analysis to obtain structured context.30 If omitted, discover context by exploring the CWD.31- **experiment_plan_path** (optional positional arg 2) — Absolute path to the full32 experiment plan. If provided, read for complete experimental methodology and design.33 If omitted, locate the experiment plan by exploring the CWD.3435## When to Use3637- Study mixes exploration and confirmation without clear boundaries38- Post-hoc hypotheses presented as pre-specified39- Many analyses run but only significant ones reported40- User invokes `/autoskillit:exp-lens-exploratory-confirmatory` or `/autoskillit:make-experiment-diag exploratory`4142## Critical Constraints4344**NEVER:**45- Modify any source code files46- Create files outside `{{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/`47- Run subagents in the background (`run_in_background: true` is prohibited)4849**ALWAYS:**50- Map the full analytic timeline — what was decided before vs. after seeing data51- Count forking paths honestly — every analysis choice is a potential fork52- Distinguish genuine exploration (hypothesis-generating) from HARKing (hypothesis-after-results)53- Flag absent preregistration as a finding without assuming bad faith54- BEFORE creating any diagram, LOAD the `/autoskillit:mermaid` skill using the Skill tool - this is MANDATORY55- If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.56- Write output to `{{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/exp_diag_exploratory_confirmatory_{YYYY-MM-DD_HHMMSS}.md`57- After writing the file, emit the structured output token as **literal plain text** with no58 markdown formatting on the token name (the adjudicator performs a regex match):5960 ```61 diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/exp_diag_exploratory_confirmatory_{...}.md62 ```6364---6566## Analysis Workflow6768### Step 0: Parse optional arguments6970If positional arg 1 (context_path) is provided and the file exists, read it to obtain71IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional72arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full73methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD74exploration for these fields if the context file supplies them.7576### Step 1: Launch Parallel Exploration Subagents7778Spawn Explore subagents to investigate:7980**Pre-specified Plans**81- Find pre-registration documents, analysis plans, hypothesis files82- Look for: preregister, analysis_plan, hypothesis, registered, protocol, spec8384**Analytic Flexibility**85- Find places where multiple analysis paths were possible86- Look for: alternatively, could_also, option, variant, subset, sensitivity, robustness8788**Selective Reporting Signals**89- Find evidence of selective reporting or cherry-picking90- Look for: not_significant, excluded, not_shown, supplementary, additional, hidden9192**Post-Hoc Rationalization**93- Find language suggesting post-hoc hypothesis generation94- Look for: we_noticed, interestingly, surprisingly, unexpectedly, upon_inspection9596**Exploration-Confirmation Separation**97- Find explicit statements about exploratory vs. confirmatory intent98- Look for: exploratory, pilot, hypothesis_generating, confirmatory, pre_specified99100### Step 2: Map Analytic Timeline101102What was decided before vs. after seeing data? Where is the exploration/confirmation boundary? Count forking paths.103104### Step 3: Analyze Boundary Integrity105106For every reported result: Was the analysis plan fixed pre-outcome? How many alternatives could have been run? Does reporting distinguish exploratory from confirmatory? Assess survivorship bias.107108### Step 4: Create the Diagram (Optional)109110**Direction:** LR (time flows left to right). Pre-data decisions → Data collection → Post-data decisions → Reporting111112### Step 5: Write Output113114Write the output to: `{{AUTOSKILLIT_TEMP}}/exp-lens-exploratory-confirmatory/exp_diag_exploratory_confirmatory_{YYYY-MM-DD_HHMMSS}.md` (relative to the current working directory)115116---117118## Pre-Diagram Checklist119120Before creating the diagram, verify:121122- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool123- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)124- [ ] Diagram will include a color legend table125126---127128## Related Skills129130- `/autoskillit:make-experiment-diag` - Parent skill131- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram132- `/autoskillit:exp-lens-severity-testing`133- `/autoskillit:exp-lens-sensitivity-robustness`