Estimand Clarity Experimental Design Lens
Philosophical Mode: Evidential
Primary Question: "What exactly is the claim?"
Focus: Effect Definition, Target Population, Outcome Specification, Comparator, Aggregation Level, Complication Handling
Arguments
/autoskillit:exp-lens-estimand-clarity [context_path] [experiment_plan_path]
- context_path (optional positional arg 1) — Absolute path to a lens context file
containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.
If provided, read this file before beginning analysis to obtain structured context.
If omitted, discover context by exploring the CWD.
- experiment_plan_path (optional positional arg 2) — Absolute path to the full
experiment plan. If provided, read for complete experimental methodology and design.
If omitted, locate the experiment plan by exploring the CWD.
When to Use
- Experiment has unclear or shifting hypotheses
- Multiple stakeholders interpret results differently
- Claims mix causal and predictive language
- User invokes
/autoskillit:exp-lens-estimand-clarity or /autoskillit:make-experiment-diag estimand
Critical Constraints
NEVER:
- Modify any source code or experiment files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Create files outside
{{AUTOSKILLIT_TEMP}}/exp-lens-estimand-clarity/
- Run subagents in the background (
run_in_background: true is prohibited)
ALWAYS:
Decompose every stated claim into formal contrast notation (Treatment A vs Treatment B on Outcome Y in Population Z)
Flag every mismatch between prose claims and code implementation
Identify the aggregation level (unit, group, time) explicitly
Document how complications (missing data, failures, exclusions) are handled
BEFORE creating any diagram, LOAD the /autoskillit:mermaid skill using the Skill tool - this is MANDATORY
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.
Write output to {{AUTOSKILLIT_TEMP}}/exp-lens-estimand-clarity/exp_diag_estimand_clarity_{YYYY-MM-DD_HHMMSS}.md
After writing the file, emit the structured output token as literal plain text with no
markdown formatting on the token name (the adjudicator performs a regex match):
diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-estimand-clarity/exp_diag_estimand_clarity_{...}.md
Analysis Workflow
Step 0: Parse optional arguments
If positional arg 1 (context_path) is provided and the file exists, read it to obtain
IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional
arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full
methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD
exploration for these fields if the context file supplies them.
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Stated Claims & Hypotheses
- Find hypothesis statements, research questions, README claims
- Look for: hypothesis, claim, goal, objective, question, we show, we demonstrate, improves, outperforms
Treatment Definition
- Find what intervention or manipulation is applied
- Look for: treatment, intervention, method, approach, condition, configuration, ablation
Outcome Definition
- Find what is measured as the result
- Look for: outcome, metric, measure, endpoint, target, response, dependent
Population & Scope
- Find what units, datasets, or contexts the claim covers
- Look for: dataset, population, sample, domain, task, benchmark, scenario, setting
Complication Handling
- Find how missing data, failures, timeouts, or exclusions are handled
- Look for: missing, exclude, timeout, fail, drop, impute, censor, incomplete
Step 2: Extract the Implicit Estimand
Answer each question from the code (not the docs):
- What is the treatment?
- What is the comparator?
- What is the outcome?
- What is the population?
- What is the time horizon?
- How are complications handled?
Step 3: Compare Claims to Implementation
Compare the explicit claims (from docs/papers) to the implicit estimand (from code). Flag mismatches between what the prose asserts and what the implementation actually measures.
CRITICAL — Analyze Claim Precision:
For every stated claim:
- Can you write it as a formal contrast (Treatment A vs Treatment B on Outcome Y in Population Z)? If not, what is ambiguous?
- Does the code measure what the prose claims?
Step 4: Create the Optional Claim-Flow Diagram
If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.
Direction: TB (claim flows from intervention through measurement to conclusion)
Small diagram: 4-6 nodes showing Treatment → Mechanism → Outcome → Claim
Node Styling:
cli class: treatment/intervention nodes
handler class: mechanism/pipeline nodes
output class: measured outcome nodes
phase class: stated claim nodes
gap class: ambiguity or mismatch between claim and measurement
Step 5: Write Output
Write the analysis to: {{AUTOSKILLIT_TEMP}}/exp-lens-estimand-clarity/exp_diag_estimand_clarity_{YYYY-MM-DD_HHMMSS}.md (relative to the current working directory)
Output Template
# Estimand Clarity Analysis: {Experiment Name}
**Lens:** Estimand Clarity (Evidential)
**Question:** What exactly is the claim?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Estimand Decomposition
| Component | Stated | Implemented | Match? |
|-----------|--------|-------------|--------|
| Treatment | {from prose} | {from code} | Yes / No / Partial |
| Comparator | {from prose} | {from code} | Yes / No / Partial |
| Outcome | {from prose} | {from code} | Yes / No / Partial |
| Population | {from prose} | {from code} | Yes / No / Partial |
| Time Horizon | {from prose} | {from code} | Yes / No / Partial |
| Complication Handling | {from prose} | {from code} | Yes / No / Partial |
## Claim Precision Assessment
| Claim | Formal Contrast | Ambiguities |
|-------|----------------|-------------|
| "{stated claim}" | Treatment A vs B on Y in Z | {list ambiguities} |
## Claim-Flow Diagram (Optional)
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart TB
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
TREATMENT["Treatment<br/>━━━━━━━━━━<br/>{intervention name}"]
MECHANISM["Mechanism<br/>━━━━━━━━━━<br/>{pipeline step}"]
OUTCOME["Measured Outcome<br/>━━━━━━━━━━<br/>{metric}"]
CLAIM["Stated Claim<br/>━━━━━━━━━━<br/>{claim text}"]
MISMATCH["Mismatch<br/>━━━━━━━━━━<br/>{ambiguity description}"]
TREATMENT --> MECHANISM
MECHANISM --> OUTCOME
OUTCOME --> CLAIM
OUTCOME -.->|"diverges"| MISMATCH
class TREATMENT cli;
class MECHANISM handler;
class OUTCOME output;
class CLAIM phase;
class MISMATCH gap;
Color Legend:
| Color |
Category |
Description |
| Dark Blue |
Treatment |
Intervention applied |
| Orange |
Mechanism |
Pipeline processing |
| Dark Teal |
Outcome |
Measured result |
| Purple |
Claim |
Stated conclusion |
| Yellow |
Mismatch |
Ambiguity or claim-code divergence |
Ambiguity Register
| # |
Ambiguity |
Location |
Severity |
Resolution Needed |
| 1 |
{description} |
{file/section} |
High / Medium / Low |
{what to clarify} |
Recommendations
- {Specific action to resolve most critical ambiguity}
- {Rewrite suggestion for vague claim}
- {Code change to align implementation with stated estimand}
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/autoskillit:make-experiment-diag` - Parent skill for lens selection
- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram
- `/autoskillit:exp-lens-causal-assumptions` - For causal structure of the stated claim
- `/autoskillit:exp-lens-measurement-validity` - For whether the outcome metric is valid
1---2name: exp-lens-estimand-clarity3categories: [exp-lens]4activate_deps: [mermaid]5description: Create Estimand Clarity experimental design analysis decomposing the implicit estimand from code vs. explicit claims from prose. Evidential lens answering "What exactly is the claim?"6hooks:7 PreToolUse:8 - matcher: "*"9 hooks:10 - type: command11 command: "echo 'Estimand Clarity Lens - Analyzing claim precision...'"12 once: true13metadata:14 source: https://skillmd.com/skills/talont-org/exp-lens-estimand-clarity15---1617# Estimand Clarity Experimental Design Lens1819**Philosophical Mode:** Evidential20**Primary Question:** "What exactly is the claim?"21**Focus:** Effect Definition, Target Population, Outcome Specification, Comparator, Aggregation Level, Complication Handling2223## Arguments2425`/autoskillit:exp-lens-estimand-clarity [context_path] [experiment_plan_path]`2627- **context_path** (optional positional arg 1) — Absolute path to a lens context file28 containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.29 If provided, read this file before beginning analysis to obtain structured context.30 If omitted, discover context by exploring the CWD.31- **experiment_plan_path** (optional positional arg 2) — Absolute path to the full32 experiment plan. If provided, read for complete experimental methodology and design.33 If omitted, locate the experiment plan by exploring the CWD.3435## When to Use3637- Experiment has unclear or shifting hypotheses38- Multiple stakeholders interpret results differently39- Claims mix causal and predictive language40- User invokes `/autoskillit:exp-lens-estimand-clarity` or `/autoskillit:make-experiment-diag estimand`4142## Critical Constraints4344**NEVER:**45- Modify any source code or experiment files46- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves47- Create files outside `{{AUTOSKILLIT_TEMP}}/exp-lens-estimand-clarity/`48- Run subagents in the background (`run_in_background: true` is prohibited)4950**ALWAYS:**51- Decompose every stated claim into formal contrast notation (Treatment A vs Treatment B on Outcome Y in Population Z)52- Flag every mismatch between prose claims and code implementation53- Identify the aggregation level (unit, group, time) explicitly54- Document how complications (missing data, failures, exclusions) are handled55- BEFORE creating any diagram, LOAD the `/autoskillit:mermaid` skill using the Skill tool - this is MANDATORY56- If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.57- Write output to `{{AUTOSKILLIT_TEMP}}/exp-lens-estimand-clarity/exp_diag_estimand_clarity_{YYYY-MM-DD_HHMMSS}.md`58- After writing the file, emit the structured output token as **literal plain text** with no59 markdown formatting on the token name (the adjudicator performs a regex match):6061 ```62 diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-estimand-clarity/exp_diag_estimand_clarity_{...}.md63 ```6465---6667## Analysis Workflow6869### Step 0: Parse optional arguments7071If positional arg 1 (context_path) is provided and the file exists, read it to obtain72IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional73arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full74methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD75exploration for these fields if the context file supplies them.7677### Step 1: Launch Parallel Exploration Subagents7879Spawn Explore subagents to investigate:8081**Stated Claims & Hypotheses**82- Find hypothesis statements, research questions, README claims83- Look for: hypothesis, claim, goal, objective, question, we show, we demonstrate, improves, outperforms8485**Treatment Definition**86- Find what intervention or manipulation is applied87- Look for: treatment, intervention, method, approach, condition, configuration, ablation8889**Outcome Definition**90- Find what is measured as the result91- Look for: outcome, metric, measure, endpoint, target, response, dependent9293**Population & Scope**94- Find what units, datasets, or contexts the claim covers95- Look for: dataset, population, sample, domain, task, benchmark, scenario, setting9697**Complication Handling**98- Find how missing data, failures, timeouts, or exclusions are handled99- Look for: missing, exclude, timeout, fail, drop, impute, censor, incomplete100101### Step 2: Extract the Implicit Estimand102103Answer each question from the code (not the docs):1041. What is the treatment?1052. What is the comparator?1063. What is the outcome?1074. What is the population?1085. What is the time horizon?1096. How are complications handled?110111### Step 3: Compare Claims to Implementation112113Compare the explicit claims (from docs/papers) to the implicit estimand (from code). Flag mismatches between what the prose asserts and what the implementation actually measures.114115**CRITICAL — Analyze Claim Precision:**116For every stated claim:117- Can you write it as a formal contrast (Treatment A vs Treatment B on Outcome Y in Population Z)? If not, what is ambiguous?118- Does the code measure what the prose claims?119120### Step 4: Create the Optional Claim-Flow Diagram121122If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.123124**Direction:** `TB` (claim flows from intervention through measurement to conclusion)125126**Small diagram: 4-6 nodes showing Treatment → Mechanism → Outcome → Claim**127128**Node Styling:**129- `cli` class: treatment/intervention nodes130- `handler` class: mechanism/pipeline nodes131- `output` class: measured outcome nodes132- `phase` class: stated claim nodes133- `gap` class: ambiguity or mismatch between claim and measurement134135### Step 5: Write Output136137Write the analysis to: `{{AUTOSKILLIT_TEMP}}/exp-lens-estimand-clarity/exp_diag_estimand_clarity_{YYYY-MM-DD_HHMMSS}.md` (relative to the current working directory)138139---140141## Output Template142143```markdown144# Estimand Clarity Analysis: {Experiment Name}145146**Lens:** Estimand Clarity (Evidential)147**Question:** What exactly is the claim?148**Date:** {YYYY-MM-DD}149**Scope:** {What was analyzed}150151## Estimand Decomposition152153| Component | Stated | Implemented | Match? |154|-----------|--------|-------------|--------|155| Treatment | {from prose} | {from code} | Yes / No / Partial |156| Comparator | {from prose} | {from code} | Yes / No / Partial |157| Outcome | {from prose} | {from code} | Yes / No / Partial |158| Population | {from prose} | {from code} | Yes / No / Partial |159| Time Horizon | {from prose} | {from code} | Yes / No / Partial |160| Complication Handling | {from prose} | {from code} | Yes / No / Partial |161162## Claim Precision Assessment163164| Claim | Formal Contrast | Ambiguities |165|-------|----------------|-------------|166| "{stated claim}" | Treatment A vs B on Y in Z | {list ambiguities} |167168## Claim-Flow Diagram (Optional)169170```mermaid171%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%172flowchart TB173 %% CLASS DEFINITIONS %%174 classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;175 classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;176 classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;177 classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;178 classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;179 classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;180 classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;181 classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;182 classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;183184 TREATMENT["Treatment<br/>━━━━━━━━━━<br/>{intervention name}"]185 MECHANISM["Mechanism<br/>━━━━━━━━━━<br/>{pipeline step}"]186 OUTCOME["Measured Outcome<br/>━━━━━━━━━━<br/>{metric}"]187 CLAIM["Stated Claim<br/>━━━━━━━━━━<br/>{claim text}"]188 MISMATCH["Mismatch<br/>━━━━━━━━━━<br/>{ambiguity description}"]189190 TREATMENT --> MECHANISM191 MECHANISM --> OUTCOME192 OUTCOME --> CLAIM193 OUTCOME -.->|"diverges"| MISMATCH194195 class TREATMENT cli;196 class MECHANISM handler;197 class OUTCOME output;198 class CLAIM phase;199 class MISMATCH gap;200```201202**Color Legend:**203| Color | Category | Description |204|-------|----------|-------------|205| Dark Blue | Treatment | Intervention applied |206| Orange | Mechanism | Pipeline processing |207| Dark Teal | Outcome | Measured result |208| Purple | Claim | Stated conclusion |209| Yellow | Mismatch | Ambiguity or claim-code divergence |210211## Ambiguity Register212213| # | Ambiguity | Location | Severity | Resolution Needed |214|---|-----------|----------|----------|-------------------|215| 1 | {description} | {file/section} | High / Medium / Low | {what to clarify} |216217## Recommendations2182191. {Specific action to resolve most critical ambiguity}2202. {Rewrite suggestion for vague claim}2213. {Code change to align implementation with stated estimand}222```223224---225226## Pre-Diagram Checklist227228Before creating the diagram, verify:229230- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool231- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)232- [ ] Diagram will include a color legend table233234---235236## Related Skills237238- `/autoskillit:make-experiment-diag` - Parent skill for lens selection239- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram240- `/autoskillit:exp-lens-causal-assumptions` - For causal structure of the stated claim241- `/autoskillit:exp-lens-measurement-validity` - For whether the outcome metric is valid