Variance Stability Experimental Design Lens
Philosophical Mode: Stability
Primary Question: "Is the signal larger than the noise?"
Focus: Run-to-Run Variance, Seed Sensitivity, Nondeterminism Sources, Confidence Intervals, Noise Floor
Arguments
/autoskillit:exp-lens-variance-stability [context_path] [experiment_plan_path]
- context_path (optional positional arg 1) — Absolute path to a lens context file
containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.
If provided, read this file before beginning analysis to obtain structured context.
If omitted, discover context by exploring the CWD.
- experiment_plan_path (optional positional arg 2) — Absolute path to the full
experiment plan. If provided, read for complete experimental methodology and design.
If omitted, locate the experiment plan by exploring the CWD.
When to Use
- ML experiments with high variance across seeds
- Systems benchmarks with environmental noise
- Results where small differences are claimed as improvements
- User invokes
/autoskillit:exp-lens-variance-stability or /autoskillit:make-experiment-diag variance
Critical Constraints
NEVER:
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Create files outside
{{AUTOSKILLIT_TEMP}}/exp-lens-variance-stability/
- Run subagents in the background (
run_in_background: true is prohibited)
ALWAYS:
Count the actual number of independent runs — single-run results must be flagged prominently
Assess whether claimed improvements exceed the observed standard deviation
Identify all sources of nondeterminism, not just random seeds
Report when confidence intervals are absent — absence is a finding, not an omission
BEFORE creating any diagram, LOAD the /autoskillit:mermaid skill using the Skill tool - this is MANDATORY
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.
Write output to {{AUTOSKILLIT_TEMP}}/exp-lens-variance-stability/exp_diag_variance_stability_{YYYY-MM-DD_HHMMSS}.md
After writing the file, emit the structured output token as literal plain text with no
markdown formatting on the token name (the adjudicator performs a regex match):
diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-variance-stability/exp_diag_variance_stability_{...}.md
Analysis Workflow
Step 0: Parse optional arguments
If positional arg 1 (context_path) is provided and the file exists, read it to obtain
IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional
arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full
methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD
exploration for these fields if the context file supplies them.
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Random Seed Management
- Find how seeds are set and varied
- Look for: seed, random_state, torch.manual_seed, np.random, set_seed, PYTHONHASHSEED
Nondeterminism Sources
- Find sources of nondeterminism beyond seeds
- Look for: cudnn, benchmark, deterministic, parallel, async, thread, race, order, nondeterministic
Multiple Run Protocol
- Find how many runs are performed per condition
- Look for: n_runs, trials, repeat, replicate, mean, std, confidence, interval, aggregate
Variance Reporting
- Find how variance is reported (if at all)
- Look for: std, stderr, confidence, interval, range, median, quartile, bootstrap, error_bar
Signal-to-Noise Assessment
- Find whether claimed improvements exceed observed variance
- Look for: significant, difference, improvement, margin, effect_size, gap, overlap
Step 2: Build Variance Profile
For each reported result:
- How many independent runs?
- What is the standard deviation across runs?
- Does the claimed improvement exceed the noise floor?
- Are confidence intervals reported?
- What sources of nondeterminism exist beyond seeds?
Build the variance profile.
Step 3: Analyze Signal vs Noise
CRITICAL — Analyze Signal vs Noise:
For every claimed improvement:
- Is the improvement magnitude larger than the run-to-run standard deviation?
- Could the ranking of methods change under reruns?
- Are the "best" results cherry-picked from multiple seeds?
Step 4: Create the Diagram
Use the mermaid skill conventions to create a stochasticity diagram with:
Direction: TB (nondeterminism sources flow down through aggregation to reported results)
Subgraphs:
- NONDETERMINISM SOURCES
- VARIANCE AGGREGATION
- REPORTED RESULTS
Node Styling:
stateNode class: Nondeterminism sources
handler class: Aggregation methods
output class: Reported results
gap class: Unreported variance or single-run results
detector class: Confidence intervals and statistical tests
cli class: Seed management
Step 5: Write Output
Write the diagram to: {{AUTOSKILLIT_TEMP}}/exp-lens-variance-stability/exp_diag_variance_stability_{YYYY-MM-DD_HHMMSS}.md (relative to the current working directory)
Output Template
# Variance Stability Analysis: {Experiment Name}
**Lens:** Variance Stability (Stability)
**Question:** Is the signal larger than the noise?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Variance Profile
| Experiment | N Runs | Mean | Std | CI | Signal > Noise? |
|------------|--------|------|-----|----|-----------------|
| {experiment} | {n} | {mean} | {std} | {CI or "Not reported"} | {Yes/No/Unclear} |
## Nondeterminism Inventory
| Source | Type | Controlled? | Impact |
|--------|------|-------------|--------|
| {source} | {seed/hardware/async/etc} | {Yes/No/Partial} | {Low/Medium/High} |
## Stochasticity Diagram
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
graph TB
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph NDSources ["NONDETERMINISM SOURCES"]
direction TB
SEED["Random Seed<br/>━━━━━━━━━━<br/>Controlled via<br/>seed management"]
HW["Hardware Variance<br/>━━━━━━━━━━<br/>GPU/CPU ordering<br/>differences"]
ASYNC["Async Operations<br/>━━━━━━━━━━<br/>Thread/process<br/>race conditions"]
end
subgraph Aggregation ["VARIANCE AGGREGATION"]
direction TB
MULTI["Multiple Runs<br/>━━━━━━━━━━<br/>N independent<br/>repetitions"]
CI["Confidence Interval<br/>━━━━━━━━━━<br/>Statistical bounds<br/>on estimates"]
end
subgraph Results ["REPORTED RESULTS"]
direction TB
RESULT["Reported Result<br/>━━━━━━━━━━<br/>Mean ± std<br/>with CI"]
SINGLE["Single-Run Result<br/>━━━━━━━━━━<br/>No variance<br/>reported"]
end
SEED --> MULTI
HW --> MULTI
ASYNC --> MULTI
MULTI --> CI
CI --> RESULT
MULTI --> SINGLE
%% CLASS ASSIGNMENTS %%
class SEED cli;
class HW,ASYNC stateNode;
class MULTI handler;
class CI detector;
class RESULT output;
class SINGLE gap;
Seed Sensitivity Assessment
| Seed |
Run Result |
Rank Among Methods |
| {seed} |
{result} |
{rank} |
Reporting Completeness Checklist
Key Findings
- {Description of whether signals exceed noise and reporting completeness}
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/autoskillit:make-experiment-diag` - Parent skill for lens selection
- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram
- `/autoskillit:exp-lens-reproducibility-artifacts` - For environment and artifact reproducibility
- `/autoskillit:exp-lens-error-budget` - For systematic error and bias analysis
1---2name: exp-lens-variance-stability3categories: [exp-lens]4activate_deps: [mermaid]5description: Create a variance analysis profile assessing whether signals exceed noise and whether results are stable across random seeds. Stability lens answering "Is the signal larger than the noise?"6hooks:7 PreToolUse:8 - matcher: "*"9 hooks:10 - type: command11 command: "echo 'Variance Stability Lens - Analyzing signal vs noise...'"12 once: true13metadata:14 source: https://skillmd.com/skills/talont-org/exp-lens-variance-stability15---1617# Variance Stability Experimental Design Lens1819**Philosophical Mode:** Stability20**Primary Question:** "Is the signal larger than the noise?"21**Focus:** Run-to-Run Variance, Seed Sensitivity, Nondeterminism Sources, Confidence Intervals, Noise Floor2223## Arguments2425`/autoskillit:exp-lens-variance-stability [context_path] [experiment_plan_path]`2627- **context_path** (optional positional arg 1) — Absolute path to a lens context file28 containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.29 If provided, read this file before beginning analysis to obtain structured context.30 If omitted, discover context by exploring the CWD.31- **experiment_plan_path** (optional positional arg 2) — Absolute path to the full32 experiment plan. If provided, read for complete experimental methodology and design.33 If omitted, locate the experiment plan by exploring the CWD.3435## When to Use3637- ML experiments with high variance across seeds38- Systems benchmarks with environmental noise39- Results where small differences are claimed as improvements40- User invokes `/autoskillit:exp-lens-variance-stability` or `/autoskillit:make-experiment-diag variance`4142## Critical Constraints4344**NEVER:**45- Modify any source code files46- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves47- Create files outside `{{AUTOSKILLIT_TEMP}}/exp-lens-variance-stability/`48- Run subagents in the background (`run_in_background: true` is prohibited)4950**ALWAYS:**51- Count the actual number of independent runs — single-run results must be flagged prominently52- Assess whether claimed improvements exceed the observed standard deviation53- Identify all sources of nondeterminism, not just random seeds54- Report when confidence intervals are absent — absence is a finding, not an omission55- BEFORE creating any diagram, LOAD the `/autoskillit:mermaid` skill using the Skill tool - this is MANDATORY56- If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.57- Write output to `{{AUTOSKILLIT_TEMP}}/exp-lens-variance-stability/exp_diag_variance_stability_{YYYY-MM-DD_HHMMSS}.md`58- After writing the file, emit the structured output token as **literal plain text** with no59 markdown formatting on the token name (the adjudicator performs a regex match):6061 ```62 diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-variance-stability/exp_diag_variance_stability_{...}.md63 ```6465---6667## Analysis Workflow6869### Step 0: Parse optional arguments7071If positional arg 1 (context_path) is provided and the file exists, read it to obtain72IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional73arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full74methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD75exploration for these fields if the context file supplies them.7677### Step 1: Launch Parallel Exploration Subagents7879Spawn Explore subagents to investigate:8081**Random Seed Management**82- Find how seeds are set and varied83- Look for: seed, random_state, torch.manual_seed, np.random, set_seed, PYTHONHASHSEED8485**Nondeterminism Sources**86- Find sources of nondeterminism beyond seeds87- Look for: cudnn, benchmark, deterministic, parallel, async, thread, race, order, nondeterministic8889**Multiple Run Protocol**90- Find how many runs are performed per condition91- Look for: n_runs, trials, repeat, replicate, mean, std, confidence, interval, aggregate9293**Variance Reporting**94- Find how variance is reported (if at all)95- Look for: std, stderr, confidence, interval, range, median, quartile, bootstrap, error_bar9697**Signal-to-Noise Assessment**98- Find whether claimed improvements exceed observed variance99- Look for: significant, difference, improvement, margin, effect_size, gap, overlap100101### Step 2: Build Variance Profile102103For each reported result:1041. How many independent runs?1052. What is the standard deviation across runs?1063. Does the claimed improvement exceed the noise floor?1074. Are confidence intervals reported?1085. What sources of nondeterminism exist beyond seeds?109110Build the variance profile.111112### Step 3: Analyze Signal vs Noise113114**CRITICAL — Analyze Signal vs Noise:**115For every claimed improvement:116- Is the improvement magnitude larger than the run-to-run standard deviation?117- Could the ranking of methods change under reruns?118- Are the "best" results cherry-picked from multiple seeds?119120### Step 4: Create the Diagram121122Use the mermaid skill conventions to create a stochasticity diagram with:123124**Direction:** `TB` (nondeterminism sources flow down through aggregation to reported results)125126**Subgraphs:**127- NONDETERMINISM SOURCES128- VARIANCE AGGREGATION129- REPORTED RESULTS130131**Node Styling:**132- `stateNode` class: Nondeterminism sources133- `handler` class: Aggregation methods134- `output` class: Reported results135- `gap` class: Unreported variance or single-run results136- `detector` class: Confidence intervals and statistical tests137- `cli` class: Seed management138139### Step 5: Write Output140141Write the diagram to: `{{AUTOSKILLIT_TEMP}}/exp-lens-variance-stability/exp_diag_variance_stability_{YYYY-MM-DD_HHMMSS}.md` (relative to the current working directory)142143---144145## Output Template146147```markdown148# Variance Stability Analysis: {Experiment Name}149150**Lens:** Variance Stability (Stability)151**Question:** Is the signal larger than the noise?152**Date:** {YYYY-MM-DD}153**Scope:** {What was analyzed}154155## Variance Profile156157| Experiment | N Runs | Mean | Std | CI | Signal > Noise? |158|------------|--------|------|-----|----|-----------------|159| {experiment} | {n} | {mean} | {std} | {CI or "Not reported"} | {Yes/No/Unclear} |160161## Nondeterminism Inventory162163| Source | Type | Controlled? | Impact |164|--------|------|-------------|--------|165| {source} | {seed/hardware/async/etc} | {Yes/No/Partial} | {Low/Medium/High} |166167## Stochasticity Diagram168169```mermaid170%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%171graph TB172 %% CLASS DEFINITIONS %%173 classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;174 classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;175 classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;176 classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;177 classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;178 classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;179 classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;180 classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;181 classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;182183 subgraph NDSources ["NONDETERMINISM SOURCES"]184 direction TB185 SEED["Random Seed<br/>━━━━━━━━━━<br/>Controlled via<br/>seed management"]186 HW["Hardware Variance<br/>━━━━━━━━━━<br/>GPU/CPU ordering<br/>differences"]187 ASYNC["Async Operations<br/>━━━━━━━━━━<br/>Thread/process<br/>race conditions"]188 end189190 subgraph Aggregation ["VARIANCE AGGREGATION"]191 direction TB192 MULTI["Multiple Runs<br/>━━━━━━━━━━<br/>N independent<br/>repetitions"]193 CI["Confidence Interval<br/>━━━━━━━━━━<br/>Statistical bounds<br/>on estimates"]194 end195196 subgraph Results ["REPORTED RESULTS"]197 direction TB198 RESULT["Reported Result<br/>━━━━━━━━━━<br/>Mean ± std<br/>with CI"]199 SINGLE["Single-Run Result<br/>━━━━━━━━━━<br/>No variance<br/>reported"]200 end201202 SEED --> MULTI203 HW --> MULTI204 ASYNC --> MULTI205 MULTI --> CI206 CI --> RESULT207 MULTI --> SINGLE208209 %% CLASS ASSIGNMENTS %%210 class SEED cli;211 class HW,ASYNC stateNode;212 class MULTI handler;213 class CI detector;214 class RESULT output;215 class SINGLE gap;216```217218## Seed Sensitivity Assessment219220| Seed | Run Result | Rank Among Methods |221|------|-----------|-------------------|222| {seed} | {result} | {rank} |223224## Reporting Completeness Checklist225226- [ ] Number of runs reported per condition227- [ ] Standard deviation or standard error reported228- [ ] Confidence intervals reported229- [ ] All seeds or seed range disclosed230- [ ] Nondeterminism sources acknowledged231232## Key Findings233234- {Description of whether signals exceed noise and reporting completeness}235```236237---238239## Pre-Diagram Checklist240241Before creating the diagram, verify:242243- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool244- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)245- [ ] Diagram will include a color legend table246247---248249## Related Skills250251- `/autoskillit:make-experiment-diag` - Parent skill for lens selection252- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram253- `/autoskillit:exp-lens-reproducibility-artifacts` - For environment and artifact reproducibility254- `/autoskillit:exp-lens-error-budget` - For systematic error and bias analysis