Pipeline Integrity Experimental Design Lens
Philosophical Mode: Integrity
Primary Question: "Could data handling create optimistic bias?"
Focus: Data Splits, Leakage Points, Preprocessing Order, Label Contamination, Pipeline Invariants
Arguments
/autoskillit:exp-lens-pipeline-integrity [context_path] [experiment_plan_path]
- context_path (optional positional arg 1) — Absolute path to a lens context file
containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.
If provided, read this file before beginning analysis to obtain structured context.
If omitted, discover context by exploring the CWD.
- experiment_plan_path (optional positional arg 2) — Absolute path to the full
experiment plan. If provided, read for complete experimental methodology and design.
If omitted, locate the experiment plan by exploring the CWD.
When to Use
- ML pipeline with train/test splits
- Preprocessing before or after splitting is ambiguous
- Feature engineering touching labels
- User invokes
/autoskillit:exp-lens-pipeline-integrity or /autoskillit:make-experiment-diag pipeline
Critical Constraints
NEVER:
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Create files outside
{{AUTOSKILLIT_TEMP}}/exp-lens-pipeline-integrity/
- Run subagents in the background (
run_in_background: true is prohibited)
ALWAYS:
Classify every pipeline stage as pre-split or post-split
Trace whether transforms are fitted on full data or train-only
Flag all label-touching feature engineering steps
Document pipeline invariants that guard against leakage
BEFORE creating any diagram, LOAD the /autoskillit:mermaid skill using the Skill tool - this is MANDATORY
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.
Write output to {{AUTOSKILLIT_TEMP}}/exp-lens-pipeline-integrity/exp_diag_pipeline_integrity_{YYYY-MM-DD_HHMMSS}.md
After writing the file, emit the structured output token as literal plain text with no
markdown formatting on the token name (the adjudicator performs a regex match):
diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-pipeline-integrity/exp_diag_pipeline_integrity_{...}.md
Analysis Workflow
Step 0: Parse optional arguments
If positional arg 1 (context_path) is provided and the file exists, read it to obtain
IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional
arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full
methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD
exploration for these fields if the context file supplies them.
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Data Loading & Sources
- Find data ingestion code, raw data paths
- Look for: load, read, fetch, dataset, csv, parquet, download
Preprocessing & Transforms
- Find normalization, encoding, imputation steps
- Look for: transform, normalize, scale, encode, impute, clean, preprocess
Split Logic
- Find train/test/validation split code
- Look for: split, train_test, fold, cross_val, stratify, group
Feature Engineering
- Find feature creation, selection, extraction
- Look for: feature, extract, select, engineer, embed, vectorize
Model Training & Evaluation
- Find training loops and evaluation metrics
- Look for: fit, train, predict, evaluate, score, metric, loss
Step 2: Map the Complete Pipeline
Map the full pipeline from raw data to reported metrics. For each stage, determine:
- What information flows in?
- What information flows out?
- Could any downstream information leak upstream?
- Classify each stage as pre-split or post-split.
Step 3: Identify Leakage Risks
CRITICAL — Analyze Leakage Direction:
For every data transformation:
- Does it use information from the full dataset (leakage risk) or only from the training partition?
- Is normalization fitted on train-only or all data?
- Are features derived from labels?
Assign a severity level (High/Medium/Low) to each leakage risk based on whether it would invalidate reported metrics.
Step 4: Create the Diagram
Use flowchart with:
Direction: LR (data flows left to right)
Subgraphs:
- RAW DATA
- PREPROCESSING
- SPLIT POINT
- TRAIN PATH
- TEST PATH
- EVALUATION
Node Styling:
cli class: Data sources
handler class: Transforms
detector class: Split point and validation gates
stateNode class: Data stores
gap class: Leakage risks
output class: Metrics and results
phase class: Model training
Edge Labels: full data, train only, test only, LEAKAGE RISK
Step 5: Write Output
Write the diagram to: {{AUTOSKILLIT_TEMP}}/exp-lens-pipeline-integrity/exp_diag_pipeline_integrity_{YYYY-MM-DD_HHMMSS}.md (relative to the current working directory)
Output Template
# Pipeline Integrity Diagram: {Experiment Name}
**Lens:** Pipeline Integrity (Integrity)
**Question:** Could data handling create optimistic bias?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Pipeline Stages
| Stage | Input | Output | Pre/Post Split | Leakage Risk? |
|-------|-------|--------|----------------|---------------|
| {stage} | {input} | {output} | {Pre/Post} | {Yes/No} |
## Pipeline Diagram
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Raw ["RAW DATA"]
SRC["Raw Dataset<br/>━━━━━━━━━━<br/>Source path<br/>N samples"]
end
subgraph Preprocessing ["PREPROCESSING"]
PREP["Normalization / Encoding<br/>━━━━━━━━━━<br/>Fitted on: full/train?"]
LEAK["Leaky Transform<br/>━━━━━━━━━━<br/>Uses full dataset"]
end
subgraph SplitPoint ["SPLIT POINT"]
SPLIT["Train/Test Split<br/>━━━━━━━━━━<br/>Stratified? Ratio?"]
end
subgraph TrainPath ["TRAIN PATH"]
TRAIN_DATA["Train Set<br/>━━━━━━━━━━<br/>N_train samples"]
MODEL["Model Training<br/>━━━━━━━━━━<br/>fit()"]
end
subgraph TestPath ["TEST PATH"]
TEST_DATA["Test Set<br/>━━━━━━━━━━<br/>N_test samples"]
end
subgraph Evaluation ["EVALUATION"]
METRIC["Reported Metric<br/>━━━━━━━━━━<br/>score / loss"]
end
%% PIPELINE FLOWS %%
SRC -->|"full data"| PREP
PREP -->|"full data"| LEAK
LEAK -.->|"LEAKAGE RISK"| METRIC
PREP -->|"full data"| SPLIT
SPLIT -->|"train only"| TRAIN_DATA
SPLIT -->|"test only"| TEST_DATA
TRAIN_DATA -->|"fit"| MODEL
MODEL -->|"predict"| TEST_DATA
TEST_DATA -->|"evaluate"| METRIC
%% CLASS ASSIGNMENTS %%
class SRC cli;
class PREP handler;
class LEAK gap;
class SPLIT detector;
class TRAIN_DATA,TEST_DATA stateNode;
class MODEL phase;
class METRIC output;
Color Legend:
| Color |
Category |
Description |
| Dark Blue |
Data Source |
Raw input datasets |
| Orange |
Transform |
Preprocessing and feature engineering steps |
| Red |
Split / Gate |
Split point and validation gates |
| Teal |
Data Store |
Partitioned data stores (train/test) |
| Purple |
Training |
Model training stages |
| Dark Teal |
Output |
Reported metrics and results |
| Amber |
Leakage Risk |
Transforms using full-dataset information |
Leakage Assessment
| Risk |
Stage |
Mechanism |
Severity |
| {risk name} |
{stage} |
{how leakage occurs} |
{High/Medium/Low} |
Pipeline Invariants
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/autoskillit:make-experiment-diag` - Parent skill for lens selection
- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram
- `/autoskillit:exp-lens-reproducibility-artifacts` - For artifact completeness audit
- `/autoskillit:exp-lens-measurement-validity` - For outcome measurement validity
1---2name: exp-lens-pipeline-integrity3categories: [exp-lens]4activate_deps: [mermaid]5description: Create Pipeline Integrity experimental design diagram showing data splits, leakage points, preprocessing order, and label contamination. Integrity lens answering "Could data handling create optimistic bias?"6hooks:7 PreToolUse:8 - matcher: "*"9 hooks:10 - type: command11 command: "echo 'Pipeline Integrity Lens - Auditing for data leakage...'"12 once: true13metadata:14 source: https://skillmd.com/skills/talont-org/exp-lens-pipeline-integrity15---1617# Pipeline Integrity Experimental Design Lens1819**Philosophical Mode:** Integrity20**Primary Question:** "Could data handling create optimistic bias?"21**Focus:** Data Splits, Leakage Points, Preprocessing Order, Label Contamination, Pipeline Invariants2223## Arguments2425`/autoskillit:exp-lens-pipeline-integrity [context_path] [experiment_plan_path]`2627- **context_path** (optional positional arg 1) — Absolute path to a lens context file28 containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.29 If provided, read this file before beginning analysis to obtain structured context.30 If omitted, discover context by exploring the CWD.31- **experiment_plan_path** (optional positional arg 2) — Absolute path to the full32 experiment plan. If provided, read for complete experimental methodology and design.33 If omitted, locate the experiment plan by exploring the CWD.3435## When to Use3637- ML pipeline with train/test splits38- Preprocessing before or after splitting is ambiguous39- Feature engineering touching labels40- User invokes `/autoskillit:exp-lens-pipeline-integrity` or `/autoskillit:make-experiment-diag pipeline`4142## Critical Constraints4344**NEVER:**45- Modify any source code files46- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves47- Create files outside `{{AUTOSKILLIT_TEMP}}/exp-lens-pipeline-integrity/`48- Run subagents in the background (`run_in_background: true` is prohibited)4950**ALWAYS:**51- Classify every pipeline stage as pre-split or post-split52- Trace whether transforms are fitted on full data or train-only53- Flag all label-touching feature engineering steps54- Document pipeline invariants that guard against leakage55- BEFORE creating any diagram, LOAD the `/autoskillit:mermaid` skill using the Skill tool - this is MANDATORY56- If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.57- Write output to `{{AUTOSKILLIT_TEMP}}/exp-lens-pipeline-integrity/exp_diag_pipeline_integrity_{YYYY-MM-DD_HHMMSS}.md`58- After writing the file, emit the structured output token as **literal plain text** with no59 markdown formatting on the token name (the adjudicator performs a regex match):6061 ```62 diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-pipeline-integrity/exp_diag_pipeline_integrity_{...}.md63 ```6465---6667## Analysis Workflow6869### Step 0: Parse optional arguments7071If positional arg 1 (context_path) is provided and the file exists, read it to obtain72IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional73arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full74methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD75exploration for these fields if the context file supplies them.7677### Step 1: Launch Parallel Exploration Subagents7879Spawn Explore subagents to investigate:8081**Data Loading & Sources**82- Find data ingestion code, raw data paths83- Look for: load, read, fetch, dataset, csv, parquet, download8485**Preprocessing & Transforms**86- Find normalization, encoding, imputation steps87- Look for: transform, normalize, scale, encode, impute, clean, preprocess8889**Split Logic**90- Find train/test/validation split code91- Look for: split, train_test, fold, cross_val, stratify, group9293**Feature Engineering**94- Find feature creation, selection, extraction95- Look for: feature, extract, select, engineer, embed, vectorize9697**Model Training & Evaluation**98- Find training loops and evaluation metrics99- Look for: fit, train, predict, evaluate, score, metric, loss100101### Step 2: Map the Complete Pipeline102103Map the full pipeline from raw data to reported metrics. For each stage, determine:104- What information flows in?105- What information flows out?106- Could any downstream information leak upstream?107- Classify each stage as pre-split or post-split.108109### Step 3: Identify Leakage Risks110111**CRITICAL — Analyze Leakage Direction:**112For every data transformation:113- Does it use information from the full dataset (leakage risk) or only from the training partition?114- Is normalization fitted on train-only or all data?115- Are features derived from labels?116117Assign a severity level (High/Medium/Low) to each leakage risk based on whether it would invalidate reported metrics.118119### Step 4: Create the Diagram120121Use flowchart with:122123**Direction:** `LR` (data flows left to right)124125**Subgraphs:**126- RAW DATA127- PREPROCESSING128- SPLIT POINT129- TRAIN PATH130- TEST PATH131- EVALUATION132133**Node Styling:**134- `cli` class: Data sources135- `handler` class: Transforms136- `detector` class: Split point and validation gates137- `stateNode` class: Data stores138- `gap` class: Leakage risks139- `output` class: Metrics and results140- `phase` class: Model training141142**Edge Labels:** full data, train only, test only, LEAKAGE RISK143144### Step 5: Write Output145146Write the diagram to: `{{AUTOSKILLIT_TEMP}}/exp-lens-pipeline-integrity/exp_diag_pipeline_integrity_{YYYY-MM-DD_HHMMSS}.md` (relative to the current working directory)147148---149150## Output Template151152```markdown153# Pipeline Integrity Diagram: {Experiment Name}154155**Lens:** Pipeline Integrity (Integrity)156**Question:** Could data handling create optimistic bias?157**Date:** {YYYY-MM-DD}158**Scope:** {What was analyzed}159160## Pipeline Stages161162| Stage | Input | Output | Pre/Post Split | Leakage Risk? |163|-------|-------|--------|----------------|---------------|164| {stage} | {input} | {output} | {Pre/Post} | {Yes/No} |165166## Pipeline Diagram167168```mermaid169%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%170flowchart LR171 %% CLASS DEFINITIONS %%172 classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;173 classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;174 classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;175 classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;176 classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;177 classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;178 classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;179 classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;180 classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;181182 subgraph Raw ["RAW DATA"]183 SRC["Raw Dataset<br/>━━━━━━━━━━<br/>Source path<br/>N samples"]184 end185186 subgraph Preprocessing ["PREPROCESSING"]187 PREP["Normalization / Encoding<br/>━━━━━━━━━━<br/>Fitted on: full/train?"]188 LEAK["Leaky Transform<br/>━━━━━━━━━━<br/>Uses full dataset"]189 end190191 subgraph SplitPoint ["SPLIT POINT"]192 SPLIT["Train/Test Split<br/>━━━━━━━━━━<br/>Stratified? Ratio?"]193 end194195 subgraph TrainPath ["TRAIN PATH"]196 TRAIN_DATA["Train Set<br/>━━━━━━━━━━<br/>N_train samples"]197 MODEL["Model Training<br/>━━━━━━━━━━<br/>fit()"]198 end199200 subgraph TestPath ["TEST PATH"]201 TEST_DATA["Test Set<br/>━━━━━━━━━━<br/>N_test samples"]202 end203204 subgraph Evaluation ["EVALUATION"]205 METRIC["Reported Metric<br/>━━━━━━━━━━<br/>score / loss"]206 end207208 %% PIPELINE FLOWS %%209 SRC -->|"full data"| PREP210 PREP -->|"full data"| LEAK211 LEAK -.->|"LEAKAGE RISK"| METRIC212 PREP -->|"full data"| SPLIT213 SPLIT -->|"train only"| TRAIN_DATA214 SPLIT -->|"test only"| TEST_DATA215 TRAIN_DATA -->|"fit"| MODEL216 MODEL -->|"predict"| TEST_DATA217 TEST_DATA -->|"evaluate"| METRIC218219 %% CLASS ASSIGNMENTS %%220 class SRC cli;221 class PREP handler;222 class LEAK gap;223 class SPLIT detector;224 class TRAIN_DATA,TEST_DATA stateNode;225 class MODEL phase;226 class METRIC output;227```228229**Color Legend:**230| Color | Category | Description |231|-------|----------|-------------|232| Dark Blue | Data Source | Raw input datasets |233| Orange | Transform | Preprocessing and feature engineering steps |234| Red | Split / Gate | Split point and validation gates |235| Teal | Data Store | Partitioned data stores (train/test) |236| Purple | Training | Model training stages |237| Dark Teal | Output | Reported metrics and results |238| Amber | Leakage Risk | Transforms using full-dataset information |239240## Leakage Assessment241242| Risk | Stage | Mechanism | Severity |243|------|-------|-----------|----------|244| {risk name} | {stage} | {how leakage occurs} | {High/Medium/Low} |245246## Pipeline Invariants247248- [ ] All scalers/encoders fitted on train partition only249- [ ] Feature selection criteria computed from train partition only250- [ ] No label information used in feature construction251- [ ] Test set never seen by any fitting step252```253254---255256## Pre-Diagram Checklist257258Before creating the diagram, verify:259260- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool261- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)262- [ ] Diagram will include a color legend table263264---265266## Related Skills267268- `/autoskillit:make-experiment-diag` - Parent skill for lens selection269- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram270- `/autoskillit:exp-lens-reproducibility-artifacts` - For artifact completeness audit271- `/autoskillit:exp-lens-measurement-validity` - For outcome measurement validity