# Comparison Design

> Design fair comparison experiments against baselines and competing methods

- Skill: `yogsoth-ai/comparison-design` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add yogsoth-ai/comparison-design`
- Raw SKILL.md: https://api.skillmd.com/api/skills/yogsoth-ai/comparison-design/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: yogsoth-ai (https://skillmd.com/u/yogsoth-ai)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/yogsoth-ai/comparison-design

---


# Strategy: Comparison Design

**Question**: How much better is our method than the baseline?

## Methodology

- **Fair Comparison Protocol** (Bouthillier 2021): Control all confounds, same compute budget, same tuning effort.
- **Multi-Baseline Comparison**: Compare against multiple baselines (SOTA, simple, ablated).
- **Multi-Dataset Evaluation**: Test across diverse datasets to avoid dataset-specific overfitting.
- **Bayesian Comparison** (Benavoli 2017): Posterior probability of superiority, not just p-values.
- **Bootstrap/Permutation Tests**: Non-parametric significance without distributional assumptions.

## Execution Flow

1. **baseline-selection** → Select appropriate baselines (SOTA, simple, oracle)
2. **metric-specification** → Define primary metric and secondary metrics
3. **sample-size-estimation** → Power analysis for detecting meaningful differences
4. **seed-protocol-design** → Ensure fair random initialization across methods
5. **environment-specification** → Lock environment to prevent confounds
6. **reproducibility-protocol** (tactic) → Ensure all results are reproducible
7. **statistical-method-selection** (tactic) → Choose Bayesian or frequentist comparison

## Budget Gate

| Comparison Scope | Baselines | Datasets | Seeds | Min Runs |
|-----------------|-----------|----------|-------|----------|
| Minimal | 1 SOTA + 1 simple | 1 | 3 | 6 |
| Standard | 2-3 baselines | 2-3 | 5 | 30-45 |
| Comprehensive | 4+ baselines | 3-5 | 5-10 | 100+ |
| Publication-ready | All relevant | 5+ | 10+ | 200+ |

<!-- BEGIN available-tables (generated) -->

## Available Tactics

Optional, no fixed order; the final leaf is always a sop.

| Tactic | When to use |
| --- | --- |
| reproducibility-protocol | Ensure experiment reproducibility through systematic environment and seed control |
| statistical-method-selection | Select appropriate statistical methods for experiment analysis |

## Available SOPs

Optional, no fixed order; the final leaf is always a sop.

| SOP | When to use |
| --- | --- |
| baseline-selection | Select appropriate baselines for experimental comparison |
| environment-specification | SOP: define complete experiment environment specification |
| metric-specification | Define experiment metrics and significance standards |
| sample-size-estimation | SOP: power analysis and required experiment count estimation |
| seed-protocol-design | SOP: design random seed strategy for reproducibility |

<!-- END available-tables (generated) -->