Comparator Construction Experimental Design Lens
Philosophical Mode: Counterfactual
Primary Question: "Is the comparator fair and relevant?"
Focus: Baseline Choice, Control Realism, Version Matching, Effort Symmetry, Baseline Drift
Arguments
/autoskillit:exp-lens-comparator-construction [context_path] [experiment_plan_path]
- context_path (optional positional arg 1) — Absolute path to a lens context file
containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.
If provided, read this file before beginning analysis to obtain structured context.
If omitted, discover context by exploring the CWD.
- experiment_plan_path (optional positional arg 2) — Absolute path to the full
experiment plan. If provided, read for complete experimental methodology and design.
If omitted, locate the experiment plan by exploring the CWD.
When to Use
- Benchmark comparisons where baseline quality is questioned
- Ablation studies needing fair controls
- Claims of improvement over prior work
- User invokes
/autoskillit:exp-lens-comparator-construction or /autoskillit:make-experiment-diag comparator
Critical Constraints
NEVER:
- Modify any source code or experiment files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Accept at face value that baselines received symmetric treatment
- Create files outside
{{AUTOSKILLIT_TEMP}}/exp-lens-comparator-construction/
- Run subagents in the background (
run_in_background: true is prohibited)
ALWAYS:
Build a fairness matrix covering all treatment-vs-comparator pairs
Check for confounding differences in implementation, tuning, data access, and compute
Assess whether each comparator is the best available alternative at the time of the experiment
Identify temporal drift in baseline relevance
BEFORE creating any diagram, LOAD the /autoskillit:mermaid skill using the Skill tool - this is MANDATORY
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.
Write output to {{AUTOSKILLIT_TEMP}}/exp-lens-comparator-construction/exp_diag_comparator_construction_{YYYY-MM-DD_HHMMSS}.md
After writing the file, emit the structured output token as literal plain text with no
markdown formatting on the token name (the adjudicator performs a regex match):
diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-comparator-construction/exp_diag_comparator_construction_{...}.md
Analysis Workflow
Step 0: Parse optional arguments
If positional arg 1 (context_path) is provided and the file exists, read it to obtain
IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional
arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full
methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD
exploration for these fields if the context file supplies them.
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Baseline/Control Definitions
- Find what the proposed method is compared against
- Look for: baseline, control, comparison, prior, state-of-the-art, vanilla, default, reference
Implementation Parity
- Find whether baselines get equal engineering effort
- Look for: reproduce, reimplement, original, paper, author, tuned, optimized, hyperparameter
Version & Environment Match
- Find whether baselines use the same software/hardware environment
- Look for: version, library, framework, gpu, hardware, environment, checkpoint
Tuning Protocol Symmetry
- Find whether hyperparameter tuning is symmetric
- Look for: tune, search, grid, optuna, sweep, budget, trials, epochs
Temporal Baseline Drift
- Find whether baselines have been updated or are stale
- Look for: date, published, year, updated, latest, deprecated, legacy
Step 2: Build the Comparator Inventory
For each comparator, assess:
- Is it the best available alternative?
- Is it given equal engineering effort?
- Is it run in the same environment?
- Is the tuning budget symmetric?
- Has it drifted since originally published?
Step 3: Construct the Fairness Matrix
CRITICAL — Analyze Counterfactual Quality:
For each treatment-vs-comparator pair:
- Does the comparison isolate the intended factor?
- Are there confounding differences in implementation, tuning, data access, or compute?
Build a fairness matrix with rows = comparators, columns = fairness dimensions.
Step 4: Create the Optional Comparison Diagram
If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.
Direction: LR (treatment and comparator flow in parallel toward evaluation)
Subgraphs: "PROPOSED METHOD", "COMPARATOR(S)", "SHARED EVALUATION"
Node Styling:
cli class: proposed method nodes
phase class: comparator method nodes
handler class: shared evaluation pipeline nodes
output class: results nodes
gap class: asymmetries flagged
detector class: parity checks
Step 5: Write Output
Write the analysis to: {{AUTOSKILLIT_TEMP}}/exp-lens-comparator-construction/exp_diag_comparator_construction_{YYYY-MM-DD_HHMMSS}.md (relative to the current working directory)
Output Template
# Comparator Construction Analysis: {Experiment Name}
**Lens:** Comparator Construction (Counterfactual)
**Question:** Is the comparator fair and relevant?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Comparator Inventory
| Comparator | Source | Reimplemented? | Same Environment? | Same Tuning Budget? |
|------------|--------|---------------|-------------------|---------------------|
| {name} | {paper/repo} | Yes / No / Partial | Yes / No | Yes / No / Unknown |
## Fairness Matrix
| Comparator | Best Available? | Equal Effort? | Same Env? | Symmetric Tuning? | Temporally Current? |
|------------|----------------|--------------|-----------|-------------------|---------------------|
| {name} | Yes / No | Yes / No | Yes / No | Yes / No | Yes / No |
## Comparison Diagram (Optional)
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
flowchart LR
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Proposed ["PROPOSED METHOD"]
METHOD["Proposed Method<br/>━━━━━━━━━━<br/>{method name}"]
end
subgraph Comparators ["COMPARATOR(S)"]
COMP1["Comparator 1<br/>━━━━━━━━━━<br/>{name}"]
COMP2["Comparator 2<br/>━━━━━━━━━━<br/>{name}"]
end
subgraph Evaluation ["SHARED EVALUATION"]
EVAL["Evaluation Pipeline<br/>━━━━━━━━━━<br/>{dataset/benchmark}"]
RESULTS["Results<br/>━━━━━━━━━━<br/>{metrics reported}"]
PARITY["Parity Check<br/>━━━━━━━━━━<br/>{asymmetry found}"]
ASYM["Asymmetry<br/>━━━━━━━━━━<br/>{description}"]
end
METHOD -->|"evaluated on"| EVAL
COMP1 -->|"evaluated on"| EVAL
COMP2 -->|"evaluated on"| EVAL
EVAL --> RESULTS
RESULTS --> PARITY
PARITY -.->|"flagged"| ASYM
class METHOD cli;
class COMP1,COMP2 phase;
class EVAL handler;
class RESULTS output;
class PARITY detector;
class ASYM gap;
Color Legend:
| Color |
Category |
Description |
| Dark Blue |
Proposed Method |
The method being evaluated |
| Purple |
Comparators |
Baselines and controls |
| Orange |
Evaluation |
Shared evaluation pipeline |
| Dark Teal |
Results |
Reported outcomes |
| Red |
Parity Checks |
Fairness verification points |
| Yellow |
Asymmetries |
Flagged unfair differences |
Asymmetry Register
| # |
Asymmetry |
Affects |
Impact Assessment |
Remediation |
| 1 |
{description} |
{comparator(s)} |
High / Medium / Low |
{how to fix} |
Recommendations
- {Most critical fairness fix — e.g., retune baseline with same budget}
- {Version alignment or environment standardization needed}
- {Additional comparator that should be included}
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/autoskillit:make-experiment-diag` - Parent skill for lens selection
- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram
- `/autoskillit:exp-lens-estimand-clarity` - For clarifying what the comparison is measuring
- `/autoskillit:exp-lens-fair-comparison` - For deeper analysis of evaluation protocol fairness
1---2name: exp-lens-comparator-construction3categories: [exp-lens]4activate_deps: [mermaid]5description: Create Comparator Construction experimental design analysis assessing whether baselines and controls are fair and relevant. Counterfactual lens answering "Is the comparator fair and relevant?"6hooks:7 PreToolUse:8 - matcher: "*"9 hooks:10 - type: command11 command: "echo 'Comparator Construction Lens - Analyzing baseline fairness...'"12 once: true13metadata:14 source: https://skillmd.com/skills/talont-org/exp-lens-comparator-construction15---1617# Comparator Construction Experimental Design Lens1819**Philosophical Mode:** Counterfactual20**Primary Question:** "Is the comparator fair and relevant?"21**Focus:** Baseline Choice, Control Realism, Version Matching, Effort Symmetry, Baseline Drift2223## Arguments2425`/autoskillit:exp-lens-comparator-construction [context_path] [experiment_plan_path]`2627- **context_path** (optional positional arg 1) — Absolute path to a lens context file28 containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.29 If provided, read this file before beginning analysis to obtain structured context.30 If omitted, discover context by exploring the CWD.31- **experiment_plan_path** (optional positional arg 2) — Absolute path to the full32 experiment plan. If provided, read for complete experimental methodology and design.33 If omitted, locate the experiment plan by exploring the CWD.3435## When to Use3637- Benchmark comparisons where baseline quality is questioned38- Ablation studies needing fair controls39- Claims of improvement over prior work40- User invokes `/autoskillit:exp-lens-comparator-construction` or `/autoskillit:make-experiment-diag comparator`4142## Critical Constraints4344**NEVER:**45- Modify any source code or experiment files46- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves47- Accept at face value that baselines received symmetric treatment48- Create files outside `{{AUTOSKILLIT_TEMP}}/exp-lens-comparator-construction/`49- Run subagents in the background (`run_in_background: true` is prohibited)5051**ALWAYS:**52- Build a fairness matrix covering all treatment-vs-comparator pairs53- Check for confounding differences in implementation, tuning, data access, and compute54- Assess whether each comparator is the best available alternative at the time of the experiment55- Identify temporal drift in baseline relevance56- BEFORE creating any diagram, LOAD the `/autoskillit:mermaid` skill using the Skill tool - this is MANDATORY57- If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.58- Write output to `{{AUTOSKILLIT_TEMP}}/exp-lens-comparator-construction/exp_diag_comparator_construction_{YYYY-MM-DD_HHMMSS}.md`59- After writing the file, emit the structured output token as **literal plain text** with no60 markdown formatting on the token name (the adjudicator performs a regex match):6162 ```63 diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-comparator-construction/exp_diag_comparator_construction_{...}.md64 ```6566---6768## Analysis Workflow6970### Step 0: Parse optional arguments7172If positional arg 1 (context_path) is provided and the file exists, read it to obtain73IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional74arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full75methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD76exploration for these fields if the context file supplies them.7778### Step 1: Launch Parallel Exploration Subagents7980Spawn Explore subagents to investigate:8182**Baseline/Control Definitions**83- Find what the proposed method is compared against84- Look for: baseline, control, comparison, prior, state-of-the-art, vanilla, default, reference8586**Implementation Parity**87- Find whether baselines get equal engineering effort88- Look for: reproduce, reimplement, original, paper, author, tuned, optimized, hyperparameter8990**Version & Environment Match**91- Find whether baselines use the same software/hardware environment92- Look for: version, library, framework, gpu, hardware, environment, checkpoint9394**Tuning Protocol Symmetry**95- Find whether hyperparameter tuning is symmetric96- Look for: tune, search, grid, optuna, sweep, budget, trials, epochs9798**Temporal Baseline Drift**99- Find whether baselines have been updated or are stale100- Look for: date, published, year, updated, latest, deprecated, legacy101102### Step 2: Build the Comparator Inventory103104For each comparator, assess:1051. Is it the best available alternative?1062. Is it given equal engineering effort?1073. Is it run in the same environment?1084. Is the tuning budget symmetric?1095. Has it drifted since originally published?110111### Step 3: Construct the Fairness Matrix112113**CRITICAL — Analyze Counterfactual Quality:**114For each treatment-vs-comparator pair:115- Does the comparison isolate the intended factor?116- Are there confounding differences in implementation, tuning, data access, or compute?117118Build a fairness matrix with rows = comparators, columns = fairness dimensions.119120### Step 4: Create the Optional Comparison Diagram121122If a diagram adds value, create a simplified flowchart. This is OPTIONAL for this hybrid lens — the tables are the primary output.123124**Direction:** `LR` (treatment and comparator flow in parallel toward evaluation)125126**Subgraphs:** "PROPOSED METHOD", "COMPARATOR(S)", "SHARED EVALUATION"127128**Node Styling:**129- `cli` class: proposed method nodes130- `phase` class: comparator method nodes131- `handler` class: shared evaluation pipeline nodes132- `output` class: results nodes133- `gap` class: asymmetries flagged134- `detector` class: parity checks135136### Step 5: Write Output137138Write the analysis to: `{{AUTOSKILLIT_TEMP}}/exp-lens-comparator-construction/exp_diag_comparator_construction_{YYYY-MM-DD_HHMMSS}.md` (relative to the current working directory)139140---141142## Output Template143144```markdown145# Comparator Construction Analysis: {Experiment Name}146147**Lens:** Comparator Construction (Counterfactual)148**Question:** Is the comparator fair and relevant?149**Date:** {YYYY-MM-DD}150**Scope:** {What was analyzed}151152## Comparator Inventory153154| Comparator | Source | Reimplemented? | Same Environment? | Same Tuning Budget? |155|------------|--------|---------------|-------------------|---------------------|156| {name} | {paper/repo} | Yes / No / Partial | Yes / No | Yes / No / Unknown |157158## Fairness Matrix159160| Comparator | Best Available? | Equal Effort? | Same Env? | Symmetric Tuning? | Temporally Current? |161|------------|----------------|--------------|-----------|-------------------|---------------------|162| {name} | Yes / No | Yes / No | Yes / No | Yes / No | Yes / No |163164## Comparison Diagram (Optional)165166```mermaid167%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%168flowchart LR169 %% CLASS DEFINITIONS %%170 classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;171 classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;172 classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;173 classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;174 classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;175 classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;176 classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;177 classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;178 classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;179180 subgraph Proposed ["PROPOSED METHOD"]181 METHOD["Proposed Method<br/>━━━━━━━━━━<br/>{method name}"]182 end183184 subgraph Comparators ["COMPARATOR(S)"]185 COMP1["Comparator 1<br/>━━━━━━━━━━<br/>{name}"]186 COMP2["Comparator 2<br/>━━━━━━━━━━<br/>{name}"]187 end188189 subgraph Evaluation ["SHARED EVALUATION"]190 EVAL["Evaluation Pipeline<br/>━━━━━━━━━━<br/>{dataset/benchmark}"]191 RESULTS["Results<br/>━━━━━━━━━━<br/>{metrics reported}"]192 PARITY["Parity Check<br/>━━━━━━━━━━<br/>{asymmetry found}"]193 ASYM["Asymmetry<br/>━━━━━━━━━━<br/>{description}"]194 end195196 METHOD -->|"evaluated on"| EVAL197 COMP1 -->|"evaluated on"| EVAL198 COMP2 -->|"evaluated on"| EVAL199 EVAL --> RESULTS200 RESULTS --> PARITY201 PARITY -.->|"flagged"| ASYM202203 class METHOD cli;204 class COMP1,COMP2 phase;205 class EVAL handler;206 class RESULTS output;207 class PARITY detector;208 class ASYM gap;209```210211**Color Legend:**212| Color | Category | Description |213|-------|----------|-------------|214| Dark Blue | Proposed Method | The method being evaluated |215| Purple | Comparators | Baselines and controls |216| Orange | Evaluation | Shared evaluation pipeline |217| Dark Teal | Results | Reported outcomes |218| Red | Parity Checks | Fairness verification points |219| Yellow | Asymmetries | Flagged unfair differences |220221## Asymmetry Register222223| # | Asymmetry | Affects | Impact Assessment | Remediation |224|---|-----------|---------|-------------------|-------------|225| 1 | {description} | {comparator(s)} | High / Medium / Low | {how to fix} |226227## Recommendations2282291. {Most critical fairness fix — e.g., retune baseline with same budget}2302. {Version alignment or environment standardization needed}2313. {Additional comparator that should be included}232```233234---235236## Pre-Diagram Checklist237238Before creating the diagram, verify:239240- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool241- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)242- [ ] Diagram will include a color legend table243244---245246## Related Skills247248- `/autoskillit:make-experiment-diag` - Parent skill for lens selection249- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram250- `/autoskillit:exp-lens-estimand-clarity` - For clarifying what the comparison is measuring251- `/autoskillit:exp-lens-fair-comparison` - For deeper analysis of evaluation protocol fairness