Reproducibility Artifacts Experimental Design Lens
Philosophical Mode: Transparency
Primary Question: "Could an independent party reproduce this?"
Focus: Run Instructions, Environment Capture, Data Availability, Determinism Controls, Audit Trail
Arguments
/autoskillit:exp-lens-reproducibility-artifacts [context_path] [experiment_plan_path]
- context_path (optional positional arg 1) — Absolute path to a lens context file
containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.
If provided, read this file before beginning analysis to obtain structured context.
If omitted, discover context by exploring the CWD.
- experiment_plan_path (optional positional arg 2) — Absolute path to the full
experiment plan. If provided, read for complete experimental methodology and design.
If omitted, locate the experiment plan by exploring the CWD.
When to Use
- Evaluating reproducibility of computational experiments
- Auditing artifact completeness
- Checking for undocumented dependencies
- User invokes
/autoskillit:exp-lens-reproducibility-artifacts or /autoskillit:make-experiment-diag reproducibility
Critical Constraints
NEVER:
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Create files outside
{{AUTOSKILLIT_TEMP}}/exp-lens-reproducibility-artifacts/
- Run subagents in the background (
run_in_background: true is prohibited)
ALWAYS:
Trace the full chain from "clone repo" to "reproduce figures"
Classify every artifact as available/unavailable and versioned/floating
Identify the weakest link in the reproduction chain
Flag all silent non-determinism risks
BEFORE creating any diagram, LOAD the /autoskillit:mermaid skill using the Skill tool - this is MANDATORY
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.
Write output to {{AUTOSKILLIT_TEMP}}/exp-lens-reproducibility-artifacts/exp_diag_reproducibility_artifacts_{YYYY-MM-DD_HHMMSS}.md
After writing the file, emit the structured output token as literal plain text with no
markdown formatting on the token name (the adjudicator performs a regex match):
diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-reproducibility-artifacts/exp_diag_reproducibility_artifacts_{...}.md
Analysis Workflow
Step 0: Parse optional arguments
If positional arg 1 (context_path) is provided and the file exists, read it to obtain
IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional
arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full
methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD
exploration for these fields if the context file supplies them.
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Environment & Dependencies
- Find dependency files, container definitions, environment setup
- Look for: requirements, Dockerfile, environment.yml, conda, pip, nix, lock
Data Provenance
- Find data download scripts, checksums, versioning
- Look for: download, checksum, hash, version, dvc, data_url, manifest
Execution Entry Points
- Find run scripts, Makefiles, workflow managers
- Look for: Makefile, run.sh, snakemake, nextflow, main, entrypoint, cli
Random Seed & Determinism
- Find seed setting, nondeterminism controls
- Look for: seed, random_state, deterministic, cudnn, PYTHONHASHSEED
Output Artifacts & Logging
- Find result storage, logging, figure generation
- Look for: save, log, output, results, figures, checkpoint, wandb, mlflow
Step 2: Map the Reproduction Chain
Map the full chain from "clone repo" to "reproduce figures." Identify each link:
- Is it documented?
- Is it automated?
- Is it deterministic?
- What is the weakest link?
Step 3: Classify Each Artifact Dependency
CRITICAL — Analyze Reproduction Chain:
For every artifact dependency:
- Is the source available (open vs gated)?
- Is the version pinned or floating?
- Is the transform deterministic?
- Could silent environment differences change results?
Assign a status of Pass, Warn, or Fail to each link in the chain based on reproducibility confidence.
Step 4: Create the Diagram
Use flowchart with:
Direction: LR (reproduction chain flows left to right)
Subgraphs:
- SOURCE CODE
- ENVIRONMENT
- DATA
- EXECUTION
- OUTPUTS
Node Styling:
cli class: Entry points and run commands
stateNode class: Versioned and pinned artifacts
handler class: Transforms and scripts
output class: Results and figures
gap class: Missing or undocumented links
detector class: Checksum and validation gates
phase class: External dependencies
Edge Labels: pinned, floating, deterministic, nondeterministic, gated
Step 5: Write Output
Write the diagram to: {{AUTOSKILLIT_TEMP}}/exp-lens-reproducibility-artifacts/exp_diag_reproducibility_artifacts_{YYYY-MM-DD_HHMMSS}.md (relative to the current working directory)
Output Template
# Reproducibility Artifacts Diagram: {Experiment Name}
**Lens:** Reproducibility Artifacts (Transparency)
**Question:** Could an independent party reproduce this?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Artifact Inventory
| Artifact | Available? | Versioned? | Deterministic? |
|----------|------------|------------|----------------|
| {artifact} | {Yes/No/Gated} | {Pinned/Floating/None} | {Yes/No/Unknown} |
## Reproduction Chain Diagram
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Source ["SOURCE CODE"]
REPO["Git Repository<br/>━━━━━━━━━━<br/>Commit hash<br/>Branch"]
ENTRY["Entry Point<br/>━━━━━━━━━━<br/>run.sh / Makefile"]
end
subgraph Env ["ENVIRONMENT"]
DEPS["Dependencies<br/>━━━━━━━━━━<br/>requirements.txt / lock"]
MISSING["Undocumented Dep<br/>━━━━━━━━━━<br/>No version pin"]
EXTDEP["External Service<br/>━━━━━━━━━━<br/>API / cloud resource"]
end
subgraph Data ["DATA"]
RAW["Raw Dataset<br/>━━━━━━━━━━<br/>Checksum available?"]
CHKSUM["Checksum Gate<br/>━━━━━━━━━━<br/>sha256 / md5"]
GATED["Gated Dataset<br/>━━━━━━━━━━<br/>Access required"]
end
subgraph Exec ["EXECUTION"]
SEED["Seed Control<br/>━━━━━━━━━━<br/>PYTHONHASHSEED<br/>random_state"]
SCRIPT["Pipeline Script<br/>━━━━━━━━━━<br/>Deterministic?"]
end
subgraph Outputs ["OUTPUTS"]
RESULTS["Results / Metrics<br/>━━━━━━━━━━<br/>Logged?"]
FIGS["Figures<br/>━━━━━━━━━━<br/>Reproducible?"]
end
%% REPRODUCTION CHAIN %%
REPO -->|"pinned"| ENTRY
ENTRY -->|"loads"| DEPS
DEPS -.->|"floating"| MISSING
MISSING -.->|"nondeterministic"| SCRIPT
EXTDEP -->|"gated"| SCRIPT
RAW -->|"verify"| CHKSUM
CHKSUM -->|"deterministic"| SCRIPT
GATED -.->|"gated"| SCRIPT
SEED -->|"controls"| SCRIPT
SCRIPT -->|"produces"| RESULTS
SCRIPT -->|"generates"| FIGS
%% CLASS ASSIGNMENTS %%
class REPO,ENTRY cli;
class DEPS,RAW stateNode;
class SCRIPT,SEED handler;
class EXTDEP phase;
class RESULTS,FIGS output;
class CHKSUM detector;
class MISSING,GATED gap;
Color Legend:
| Color |
Category |
Description |
| Dark Blue |
Entry Point |
Run commands and source code |
| Teal |
Versioned Artifact |
Pinned dependencies and checksummed data |
| Orange |
Transform / Script |
Pipeline scripts and execution steps |
| Purple |
External Dependency |
External services and APIs |
| Dark Teal |
Output |
Results, metrics, and figures |
| Red |
Validation Gate |
Checksum and integrity checks |
| Amber |
Missing Link |
Undocumented or gated dependencies |
Reproduction Checklist
Step-by-step instructions with pass/fail status:
Weakest Links
| Link |
Issue |
Severity |
Recommendation |
| {link} |
{undocumented/gated/floating/nondeterministic} |
{High/Medium/Low} |
{action} |
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/autoskillit:make-experiment-diag` - Parent skill for lens selection
- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram
- `/autoskillit:exp-lens-pipeline-integrity` - For data leakage audit
- `/autoskillit:exp-lens-variance-stability` - For result stability across seeds
1---2name: exp-lens-reproducibility-artifacts3categories: [exp-lens]4activate_deps: [mermaid]5description: Create Reproducibility Artifacts experimental design diagram showing run instructions, environment capture, data availability, determinism controls, and audit trail. Transparency lens answering "Could an independent party reproduce this?"6hooks:7 PreToolUse:8 - matcher: "*"9 hooks:10 - type: command11 command: "echo 'Reproducibility Artifacts Lens - Auditing reproduction chain...'"12 once: true13metadata:14 source: https://skillmd.com/skills/talont-org/exp-lens-reproducibility-artifacts15---1617# Reproducibility Artifacts Experimental Design Lens1819**Philosophical Mode:** Transparency20**Primary Question:** "Could an independent party reproduce this?"21**Focus:** Run Instructions, Environment Capture, Data Availability, Determinism Controls, Audit Trail2223## Arguments2425`/autoskillit:exp-lens-reproducibility-artifacts [context_path] [experiment_plan_path]`2627- **context_path** (optional positional arg 1) — Absolute path to a lens context file28 containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.29 If provided, read this file before beginning analysis to obtain structured context.30 If omitted, discover context by exploring the CWD.31- **experiment_plan_path** (optional positional arg 2) — Absolute path to the full32 experiment plan. If provided, read for complete experimental methodology and design.33 If omitted, locate the experiment plan by exploring the CWD.3435## When to Use3637- Evaluating reproducibility of computational experiments38- Auditing artifact completeness39- Checking for undocumented dependencies40- User invokes `/autoskillit:exp-lens-reproducibility-artifacts` or `/autoskillit:make-experiment-diag reproducibility`4142## Critical Constraints4344**NEVER:**45- Modify any source code files46- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves47- Create files outside `{{AUTOSKILLIT_TEMP}}/exp-lens-reproducibility-artifacts/`48- Run subagents in the background (`run_in_background: true` is prohibited)4950**ALWAYS:**51- Trace the full chain from "clone repo" to "reproduce figures"52- Classify every artifact as available/unavailable and versioned/floating53- Identify the weakest link in the reproduction chain54- Flag all silent non-determinism risks55- BEFORE creating any diagram, LOAD the `/autoskillit:mermaid` skill using the Skill tool - this is MANDATORY56- If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.57- Write output to `{{AUTOSKILLIT_TEMP}}/exp-lens-reproducibility-artifacts/exp_diag_reproducibility_artifacts_{YYYY-MM-DD_HHMMSS}.md`58- After writing the file, emit the structured output token as **literal plain text** with no59 markdown formatting on the token name (the adjudicator performs a regex match):6061 ```62 diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-reproducibility-artifacts/exp_diag_reproducibility_artifacts_{...}.md63 ```6465---6667## Analysis Workflow6869### Step 0: Parse optional arguments7071If positional arg 1 (context_path) is provided and the file exists, read it to obtain72IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional73arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full74methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD75exploration for these fields if the context file supplies them.7677### Step 1: Launch Parallel Exploration Subagents7879Spawn Explore subagents to investigate:8081**Environment & Dependencies**82- Find dependency files, container definitions, environment setup83- Look for: requirements, Dockerfile, environment.yml, conda, pip, nix, lock8485**Data Provenance**86- Find data download scripts, checksums, versioning87- Look for: download, checksum, hash, version, dvc, data_url, manifest8889**Execution Entry Points**90- Find run scripts, Makefiles, workflow managers91- Look for: Makefile, run.sh, snakemake, nextflow, main, entrypoint, cli9293**Random Seed & Determinism**94- Find seed setting, nondeterminism controls95- Look for: seed, random_state, deterministic, cudnn, PYTHONHASHSEED9697**Output Artifacts & Logging**98- Find result storage, logging, figure generation99- Look for: save, log, output, results, figures, checkpoint, wandb, mlflow100101### Step 2: Map the Reproduction Chain102103Map the full chain from "clone repo" to "reproduce figures." Identify each link:104- Is it documented?105- Is it automated?106- Is it deterministic?107- What is the weakest link?108109### Step 3: Classify Each Artifact Dependency110111**CRITICAL — Analyze Reproduction Chain:**112For every artifact dependency:113- Is the source available (open vs gated)?114- Is the version pinned or floating?115- Is the transform deterministic?116- Could silent environment differences change results?117118Assign a status of Pass, Warn, or Fail to each link in the chain based on reproducibility confidence.119120### Step 4: Create the Diagram121122Use flowchart with:123124**Direction:** `LR` (reproduction chain flows left to right)125126**Subgraphs:**127- SOURCE CODE128- ENVIRONMENT129- DATA130- EXECUTION131- OUTPUTS132133**Node Styling:**134- `cli` class: Entry points and run commands135- `stateNode` class: Versioned and pinned artifacts136- `handler` class: Transforms and scripts137- `output` class: Results and figures138- `gap` class: Missing or undocumented links139- `detector` class: Checksum and validation gates140- `phase` class: External dependencies141142**Edge Labels:** pinned, floating, deterministic, nondeterministic, gated143144### Step 5: Write Output145146Write the diagram to: `{{AUTOSKILLIT_TEMP}}/exp-lens-reproducibility-artifacts/exp_diag_reproducibility_artifacts_{YYYY-MM-DD_HHMMSS}.md` (relative to the current working directory)147148---149150## Output Template151152```markdown153# Reproducibility Artifacts Diagram: {Experiment Name}154155**Lens:** Reproducibility Artifacts (Transparency)156**Question:** Could an independent party reproduce this?157**Date:** {YYYY-MM-DD}158**Scope:** {What was analyzed}159160## Artifact Inventory161162| Artifact | Available? | Versioned? | Deterministic? |163|----------|------------|------------|----------------|164| {artifact} | {Yes/No/Gated} | {Pinned/Floating/None} | {Yes/No/Unknown} |165166## Reproduction Chain Diagram167168```mermaid169%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%170flowchart LR171 %% CLASS DEFINITIONS %%172 classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;173 classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;174 classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;175 classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;176 classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;177 classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;178 classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;179 classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;180 classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;181182 subgraph Source ["SOURCE CODE"]183 REPO["Git Repository<br/>━━━━━━━━━━<br/>Commit hash<br/>Branch"]184 ENTRY["Entry Point<br/>━━━━━━━━━━<br/>run.sh / Makefile"]185 end186187 subgraph Env ["ENVIRONMENT"]188 DEPS["Dependencies<br/>━━━━━━━━━━<br/>requirements.txt / lock"]189 MISSING["Undocumented Dep<br/>━━━━━━━━━━<br/>No version pin"]190 EXTDEP["External Service<br/>━━━━━━━━━━<br/>API / cloud resource"]191 end192193 subgraph Data ["DATA"]194 RAW["Raw Dataset<br/>━━━━━━━━━━<br/>Checksum available?"]195 CHKSUM["Checksum Gate<br/>━━━━━━━━━━<br/>sha256 / md5"]196 GATED["Gated Dataset<br/>━━━━━━━━━━<br/>Access required"]197 end198199 subgraph Exec ["EXECUTION"]200 SEED["Seed Control<br/>━━━━━━━━━━<br/>PYTHONHASHSEED<br/>random_state"]201 SCRIPT["Pipeline Script<br/>━━━━━━━━━━<br/>Deterministic?"]202 end203204 subgraph Outputs ["OUTPUTS"]205 RESULTS["Results / Metrics<br/>━━━━━━━━━━<br/>Logged?"]206 FIGS["Figures<br/>━━━━━━━━━━<br/>Reproducible?"]207 end208209 %% REPRODUCTION CHAIN %%210 REPO -->|"pinned"| ENTRY211 ENTRY -->|"loads"| DEPS212 DEPS -.->|"floating"| MISSING213 MISSING -.->|"nondeterministic"| SCRIPT214 EXTDEP -->|"gated"| SCRIPT215 RAW -->|"verify"| CHKSUM216 CHKSUM -->|"deterministic"| SCRIPT217 GATED -.->|"gated"| SCRIPT218 SEED -->|"controls"| SCRIPT219 SCRIPT -->|"produces"| RESULTS220 SCRIPT -->|"generates"| FIGS221222 %% CLASS ASSIGNMENTS %%223 class REPO,ENTRY cli;224 class DEPS,RAW stateNode;225 class SCRIPT,SEED handler;226 class EXTDEP phase;227 class RESULTS,FIGS output;228 class CHKSUM detector;229 class MISSING,GATED gap;230```231232**Color Legend:**233| Color | Category | Description |234|-------|----------|-------------|235| Dark Blue | Entry Point | Run commands and source code |236| Teal | Versioned Artifact | Pinned dependencies and checksummed data |237| Orange | Transform / Script | Pipeline scripts and execution steps |238| Purple | External Dependency | External services and APIs |239| Dark Teal | Output | Results, metrics, and figures |240| Red | Validation Gate | Checksum and integrity checks |241| Amber | Missing Link | Undocumented or gated dependencies |242243## Reproduction Checklist244245Step-by-step instructions with pass/fail status:246247- [ ] Clone repository at pinned commit248- [ ] Reproduce environment from lock/container file249- [ ] Download data and verify checksums250- [ ] Set all random seeds as documented251- [ ] Execute pipeline via documented entry point252- [ ] Compare output metrics/figures to reported values253254## Weakest Links255256| Link | Issue | Severity | Recommendation |257|------|-------|----------|----------------|258| {link} | {undocumented/gated/floating/nondeterministic} | {High/Medium/Low} | {action} |259```260261---262263## Pre-Diagram Checklist264265Before creating the diagram, verify:266267- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool268- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)269- [ ] Diagram will include a color legend table270271---272273## Related Skills274275- `/autoskillit:make-experiment-diag` - Parent skill for lens selection276- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram277- `/autoskillit:exp-lens-pipeline-integrity` - For data leakage audit278- `/autoskillit:exp-lens-variance-stability` - For result stability across seeds