Fair Comparison Experimental Design Lens
Philosophical Mode: Fairness
Primary Question: "Are alternatives compared under symmetric constraints?"
Focus: Compute Budget Symmetry, Tuning Protocol Parity, Data Access Equality, Engineering Effort Balance, Winner's Curse
Arguments
/autoskillit:exp-lens-fair-comparison [context_path] [experiment_plan_path]
- context_path (optional positional arg 1) — Absolute path to a lens context file
containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.
If provided, read this file before beginning analysis to obtain structured context.
If omitted, discover context by exploring the CWD.
- experiment_plan_path (optional positional arg 2) — Absolute path to the full
experiment plan. If provided, read for complete experimental methodology and design.
If omitted, locate the experiment plan by exploring the CWD.
When to Use
- Method comparisons where tuning effort differs
- Benchmark results where compute budgets are unequal
- Claims of SOTA that may reflect process differences
- User invokes
/autoskillit:exp-lens-fair-comparison or /autoskillit:make-experiment-diag fairness
Critical Constraints
NEVER:
- Modify any source code files
- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves
- Create files outside
{{AUTOSKILLIT_TEMP}}/exp-lens-fair-comparison/
- Run subagents in the background (
run_in_background: true is prohibited)
ALWAYS:
Build the full symmetry matrix — every method against every resource dimension
Attribute improvements to method vs. process — both deserve accounting
Flag undisclosed compute or tuning as a finding, not an assumption
Assess the winner's curse: did the proposed method benefit from more selection pressure?
BEFORE creating any diagram, LOAD the /autoskillit:mermaid skill using the Skill tool - this is MANDATORY
If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.
Write output to {{AUTOSKILLIT_TEMP}}/exp-lens-fair-comparison/exp_diag_fair_comparison_{YYYY-MM-DD_HHMMSS}.md
After writing the file, emit the structured output token as literal plain text with no
markdown formatting on the token name (the adjudicator performs a regex match):
diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-fair-comparison/exp_diag_fair_comparison_{...}.md
Analysis Workflow
Step 0: Parse optional arguments
If positional arg 1 (context_path) is provided and the file exists, read it to obtain
IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional
arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full
methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD
exploration for these fields if the context file supplies them.
Step 1: Launch Parallel Exploration Subagents
Spawn Explore subagents to investigate:
Compute & Resource Allocation
- Find compute resources used per method
- Look for: gpu, tpu, hours, cost, memory, flops, compute_budget, machine, cluster
Tuning Protocol per Method
- Find tuning procedures for each compared method
- Look for: grid_search, optuna, bayesian_opt, hyperband, tune, sweep, trials, budget, early_stop
Data Access & Preprocessing
- Find whether all methods use the same data pipeline
- Look for: data_augmentation, preprocessing, feature, embedding, pretrained, extra_data, auxiliary
Engineering Effort Indicators
- Find differential engineering investment
- Look for: custom, specialized, trick, hack, ensemble, post_process, calibrate, threshold_tune
Reporting Completeness
- Find whether resource usage is disclosed
- Look for: report, disclose, computational_cost, wall_time, parameter_count, training_time
Step 2: Build Symmetry Matrix
Build the symmetry matrix: rows = methods compared, columns = resource dimensions (compute, tuning, data, engineering, disclosure).
For each cell:
- Is the allocation symmetric?
- If not, does the asymmetry favor the proposed method?
- Estimate the magnitude of bias from each asymmetry.
Step 3: Analyze Effort Attribution
CRITICAL — Analyze Effort Attribution:
For every claimed improvement:
- What fraction of the improvement can be attributed to the method itself vs. differential engineering effort, tuning budget, or data access?
Step 4: Create the Diagram
Use the mermaid skill conventions to create a symmetry diagram with:
Direction: LR (methods flow through resource allocation to evaluation)
Subgraphs:
- METHODS
- RESOURCE ALLOCATION
- EVALUATION
Node Styling:
cli class: Proposed method
phase class: Comparator methods
handler class: Shared resources
gap class: Asymmetric resources
detector class: Symmetry checks
output class: Results
Step 5: Write Output
Write the diagram to: {{AUTOSKILLIT_TEMP}}/exp-lens-fair-comparison/exp_diag_fair_comparison_{YYYY-MM-DD_HHMMSS}.md (relative to the current working directory)
Output Template
# Fair Comparison Analysis: {Experiment Name}
**Lens:** Fair Comparison (Fairness)
**Question:** Are alternatives compared under symmetric constraints?
**Date:** {YYYY-MM-DD}
**Scope:** {What was analyzed}
## Symmetry Matrix
| Method | Compute | Tuning Budget | Data Access | Engineering | Disclosure |
|--------|---------|---------------|-------------|-------------|------------|
| {proposed method} | {allocation} | {budget} | {access} | {effort} | {disclosed?} |
| {comparator} | {allocation} | {budget} | {access} | {effort} | {disclosed?} |
## Resource Disclosure
| Resource Type | Proposed Method | Comparators | Symmetric? |
|---------------|-----------------|-------------|------------|
| {GPU hours} | {value} | {value} | {Yes/No} |
| {Tuning trials} | {value} | {value} | {Yes/No} |
| {Extra data} | {value} | {value} | {Yes/No} |
## Symmetry Diagram
```mermaid
%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%
graph LR
%% CLASS DEFINITIONS %%
classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;
classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;
classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;
classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;
classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;
classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;
classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;
classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;
subgraph Methods ["METHODS"]
direction TB
PROP["Proposed Method<br/>━━━━━━━━━━<br/>Method under<br/>evaluation"]
COMP["Comparator Methods<br/>━━━━━━━━━━<br/>Baseline and<br/>prior work"]
end
subgraph Resources ["RESOURCE ALLOCATION"]
direction TB
SHARED["Shared Resources<br/>━━━━━━━━━━<br/>Same data, same<br/>evaluation protocol"]
ASYM["Asymmetric Resources<br/>━━━━━━━━━━<br/>Differential compute<br/>or tuning budget"]
CHECK["Symmetry Check<br/>━━━━━━━━━━<br/>Verify parity<br/>across methods"]
end
subgraph Eval ["EVALUATION"]
direction TB
RESULT["Results<br/>━━━━━━━━━━<br/>Reported<br/>performance"]
end
PROP --> SHARED
COMP --> SHARED
PROP --> ASYM
SHARED --> CHECK
ASYM --> CHECK
CHECK --> RESULT
%% CLASS ASSIGNMENTS %%
class PROP cli;
class COMP phase;
class SHARED handler;
class ASYM gap;
class CHECK detector;
class RESULT output;
Winner's Curse Assessment
| Factor |
Proposed Method Advantage |
Impact on Claimed Improvement |
| {tuning trials} |
{advantage} |
{estimated impact} |
| {engineering tricks} |
{advantage} |
{estimated impact} |
Process-vs-Method Attribution Analysis
- Method contribution: {estimated fraction}
- Tuning contribution: {estimated fraction}
- Engineering contribution: {estimated fraction}
- Data access contribution: {estimated fraction}
Key Findings
- {Description of most significant asymmetries and their impact on claimed improvements}
---
## Pre-Diagram Checklist
Before creating the diagram, verify:
- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool
- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)
- [ ] Diagram will include a color legend table
---
## Related Skills
- `/autoskillit:make-experiment-diag` - Parent skill for lens selection
- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram
- `/autoskillit:exp-lens-comparator-construction` - For baseline selection and construction adequacy
- `/autoskillit:exp-lens-sensitivity-robustness` - For sensitivity analysis across conditions
1---2name: exp-lens-fair-comparison3categories: [exp-lens]4activate_deps: [mermaid]5description: Create a comparison fairness matrix assessing whether alternatives are evaluated under symmetric constraints. Fairness lens answering "Are alternatives compared under symmetric constraints?"6hooks:7 PreToolUse:8 - matcher: "*"9 hooks:10 - type: command11 command: "echo 'Fair Comparison Lens - Analyzing symmetry of constraints...'"12 once: true13metadata:14 source: https://skillmd.com/skills/talont-org/exp-lens-fair-comparison15---1617# Fair Comparison Experimental Design Lens1819**Philosophical Mode:** Fairness20**Primary Question:** "Are alternatives compared under symmetric constraints?"21**Focus:** Compute Budget Symmetry, Tuning Protocol Parity, Data Access Equality, Engineering Effort Balance, Winner's Curse2223## Arguments2425`/autoskillit:exp-lens-fair-comparison [context_path] [experiment_plan_path]`2627- **context_path** (optional positional arg 1) — Absolute path to a lens context file28 containing IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria.29 If provided, read this file before beginning analysis to obtain structured context.30 If omitted, discover context by exploring the CWD.31- **experiment_plan_path** (optional positional arg 2) — Absolute path to the full32 experiment plan. If provided, read for complete experimental methodology and design.33 If omitted, locate the experiment plan by exploring the CWD.3435## When to Use3637- Method comparisons where tuning effort differs38- Benchmark results where compute budgets are unequal39- Claims of SOTA that may reflect process differences40- User invokes `/autoskillit:exp-lens-fair-comparison` or `/autoskillit:make-experiment-diag fairness`4142## Critical Constraints4344**NEVER:**45- Modify any source code files46- Do not litter the codebase with useless comments, TODO markers, or explanatory annotations — the skill output and diagram speak for themselves47- Create files outside `{{AUTOSKILLIT_TEMP}}/exp-lens-fair-comparison/`48- Run subagents in the background (`run_in_background: true` is prohibited)4950**ALWAYS:**51- Build the full symmetry matrix — every method against every resource dimension52- Attribute improvements to method vs. process — both deserve accounting53- Flag undisclosed compute or tuning as a finding, not an assumption54- Assess the winner's curse: did the proposed method benefit from more selection pressure?55- BEFORE creating any diagram, LOAD the `/autoskillit:mermaid` skill using the Skill tool - this is MANDATORY56- If the Skill tool cannot be used (disable-model-invocation) or refuses this invocation, do NOT proceed with diagram creation. Abort this step and omit the diagram from output.57- Write output to `{{AUTOSKILLIT_TEMP}}/exp-lens-fair-comparison/exp_diag_fair_comparison_{YYYY-MM-DD_HHMMSS}.md`58- After writing the file, emit the structured output token as **literal plain text** with no59 markdown formatting on the token name (the adjudicator performs a regex match):6061 ```62 diagram_path = /absolute/path/to/{{AUTOSKILLIT_TEMP}}/exp-lens-fair-comparison/exp_diag_fair_comparison_{...}.md63 ```6465---6667## Analysis Workflow6869### Step 0: Parse optional arguments7071If positional arg 1 (context_path) is provided and the file exists, read it to obtain72IV/DV tables, H0/H1 hypotheses, controlled variables, and success criteria. If positional73arg 2 (experiment_plan_path) is provided and exists, read the experiment plan for full74methodology. Use this structured context as the foundation for Steps 1-5; skip the CWD75exploration for these fields if the context file supplies them.7677### Step 1: Launch Parallel Exploration Subagents7879Spawn Explore subagents to investigate:8081**Compute & Resource Allocation**82- Find compute resources used per method83- Look for: gpu, tpu, hours, cost, memory, flops, compute_budget, machine, cluster8485**Tuning Protocol per Method**86- Find tuning procedures for each compared method87- Look for: grid_search, optuna, bayesian_opt, hyperband, tune, sweep, trials, budget, early_stop8889**Data Access & Preprocessing**90- Find whether all methods use the same data pipeline91- Look for: data_augmentation, preprocessing, feature, embedding, pretrained, extra_data, auxiliary9293**Engineering Effort Indicators**94- Find differential engineering investment95- Look for: custom, specialized, trick, hack, ensemble, post_process, calibrate, threshold_tune9697**Reporting Completeness**98- Find whether resource usage is disclosed99- Look for: report, disclose, computational_cost, wall_time, parameter_count, training_time100101### Step 2: Build Symmetry Matrix102103Build the symmetry matrix: rows = methods compared, columns = resource dimensions (compute, tuning, data, engineering, disclosure).104105For each cell:106- Is the allocation symmetric?107- If not, does the asymmetry favor the proposed method?108- Estimate the magnitude of bias from each asymmetry.109110### Step 3: Analyze Effort Attribution111112**CRITICAL — Analyze Effort Attribution:**113For every claimed improvement:114- What fraction of the improvement can be attributed to the method itself vs. differential engineering effort, tuning budget, or data access?115116### Step 4: Create the Diagram117118Use the mermaid skill conventions to create a symmetry diagram with:119120**Direction:** `LR` (methods flow through resource allocation to evaluation)121122**Subgraphs:**123- METHODS124- RESOURCE ALLOCATION125- EVALUATION126127**Node Styling:**128- `cli` class: Proposed method129- `phase` class: Comparator methods130- `handler` class: Shared resources131- `gap` class: Asymmetric resources132- `detector` class: Symmetry checks133- `output` class: Results134135### Step 5: Write Output136137Write the diagram to: `{{AUTOSKILLIT_TEMP}}/exp-lens-fair-comparison/exp_diag_fair_comparison_{YYYY-MM-DD_HHMMSS}.md` (relative to the current working directory)138139---140141## Output Template142143```markdown144# Fair Comparison Analysis: {Experiment Name}145146**Lens:** Fair Comparison (Fairness)147**Question:** Are alternatives compared under symmetric constraints?148**Date:** {YYYY-MM-DD}149**Scope:** {What was analyzed}150151## Symmetry Matrix152153| Method | Compute | Tuning Budget | Data Access | Engineering | Disclosure |154|--------|---------|---------------|-------------|-------------|------------|155| {proposed method} | {allocation} | {budget} | {access} | {effort} | {disclosed?} |156| {comparator} | {allocation} | {budget} | {access} | {effort} | {disclosed?} |157158## Resource Disclosure159160| Resource Type | Proposed Method | Comparators | Symmetric? |161|---------------|-----------------|-------------|------------|162| {GPU hours} | {value} | {value} | {Yes/No} |163| {Tuning trials} | {value} | {value} | {Yes/No} |164| {Extra data} | {value} | {value} | {Yes/No} |165166## Symmetry Diagram167168```mermaid169%%{init: {'flowchart': {'nodeSpacing': 50, 'rankSpacing': 60, 'curve': 'basis'}}}%%170graph LR171 %% CLASS DEFINITIONS %%172 classDef cli fill:#1a237e,stroke:#7986cb,stroke-width:2px,color:#fff;173 classDef stateNode fill:#004d40,stroke:#4db6ac,stroke-width:2px,color:#fff;174 classDef handler fill:#e65100,stroke:#ffb74d,stroke-width:2px,color:#fff;175 classDef phase fill:#6a1b9a,stroke:#ba68c8,stroke-width:2px,color:#fff;176 classDef newComponent fill:#2e7d32,stroke:#81c784,stroke-width:2px,color:#fff;177 classDef output fill:#00695c,stroke:#4db6ac,stroke-width:2px,color:#fff;178 classDef detector fill:#b71c1c,stroke:#ef5350,stroke-width:2px,color:#fff;179 classDef gap fill:#ff6f00,stroke:#ffa726,stroke-width:2px,color:#000;180 classDef integration fill:#c62828,stroke:#ef9a9a,stroke-width:2px,color:#fff;181182 subgraph Methods ["METHODS"]183 direction TB184 PROP["Proposed Method<br/>━━━━━━━━━━<br/>Method under<br/>evaluation"]185 COMP["Comparator Methods<br/>━━━━━━━━━━<br/>Baseline and<br/>prior work"]186 end187188 subgraph Resources ["RESOURCE ALLOCATION"]189 direction TB190 SHARED["Shared Resources<br/>━━━━━━━━━━<br/>Same data, same<br/>evaluation protocol"]191 ASYM["Asymmetric Resources<br/>━━━━━━━━━━<br/>Differential compute<br/>or tuning budget"]192 CHECK["Symmetry Check<br/>━━━━━━━━━━<br/>Verify parity<br/>across methods"]193 end194195 subgraph Eval ["EVALUATION"]196 direction TB197 RESULT["Results<br/>━━━━━━━━━━<br/>Reported<br/>performance"]198 end199200 PROP --> SHARED201 COMP --> SHARED202 PROP --> ASYM203 SHARED --> CHECK204 ASYM --> CHECK205 CHECK --> RESULT206207 %% CLASS ASSIGNMENTS %%208 class PROP cli;209 class COMP phase;210 class SHARED handler;211 class ASYM gap;212 class CHECK detector;213 class RESULT output;214```215216## Winner's Curse Assessment217218| Factor | Proposed Method Advantage | Impact on Claimed Improvement |219|--------|--------------------------|-------------------------------|220| {tuning trials} | {advantage} | {estimated impact} |221| {engineering tricks} | {advantage} | {estimated impact} |222223## Process-vs-Method Attribution Analysis224225- Method contribution: {estimated fraction}226- Tuning contribution: {estimated fraction}227- Engineering contribution: {estimated fraction}228- Data access contribution: {estimated fraction}229230## Key Findings231232- {Description of most significant asymmetries and their impact on claimed improvements}233```234235---236237## Pre-Diagram Checklist238239Before creating the diagram, verify:240241- [ ] LOADED `/autoskillit:mermaid` skill using the Skill tool242- [ ] Using ONLY classDef styles from the mermaid skill (no invented colors)243- [ ] Diagram will include a color legend table244245---246247## Related Skills248249- `/autoskillit:make-experiment-diag` - Parent skill for lens selection250- `/autoskillit:mermaid` - MUST BE LOADED before creating diagram251- `/autoskillit:exp-lens-comparator-construction` - For baseline selection and construction adequacy252- `/autoskillit:exp-lens-sensitivity-robustness` - For sensitivity analysis across conditions