establish-empirical-baseline
Purpose
Establish a fair empirical baseline by inventorying methods, extracting comparable performance, normalizing conditions/compute, checking discrepancies, and estimating progress/headroom.
Input contract
required: [method_records, benchmark_or_task, performance_measure]
optional: [historical_series, compute_metadata, condition_schema]
constraints: [comparability fields and source provenance required]
Execution protocol
Do not perform called SOP operations inline; each loaded SOP owns its contract and thresholds.
- You MUST load skill
inventory-reference-items to inventory methods and comparison records.
- You MUST load skill
extract-evidence-record to extract performance evidence. You MUST load skill audit-reporting-quality to audit missing or ambiguous reporting. You MUST load skill normalize-comparison-scale to normalize units, data, compute, and evaluation protocol.
- You MUST load skill
detect-performance-discrepancy to detect comparison discrepancies. You MUST load skill estimate-performance-headroom to estimate headroom. You MUST load skill analyze-temporal-trajectory to analyze progress and leaderboard dynamics. You MUST load skill check-dominance to identify dominated and incomparable records.
- Synthesize the baseline with uncertainty and known incomparable records.
If the resulting methods or baselines require explicit comparative selection, consider
rank-candidates as the next tactic.
Output contract
produces: [method_inventory, normalized_baseline, discrepancy_report, progress_curve, headroom_estimate]
delta_fields: [findings, evidence_updates, uncertainties, decisions, open_questions]
Thresholds and quality gates
- Baseline acquisition gates are relative to a declared eligible universe: record numerator, denominator, batch increment, stopping reason, and source references for each ratio.
method-inventory: method coverage ratio reaches a justified floor over the eligible method universe.
performance-extraction: comparable-record coverage ratio reaches a justified floor over eligible records.
condition-standardization: complete condition-vector ratio reaches a justified floor over comparable records.
discrepancy-analysis: score-pair coverage ratio reaches a justified floor over eligible comparison pairs.
progress-quantification: historical-time coverage and independent-source ratio reach justified floors; stop when added periods no longer change the trajectory conclusion.
- Across modes, report marginal information gain and saturation state when added records or periods no longer change the baseline conclusion.
- Normalization must expose condition, compute, metric, and unit transformations.
Failure and counterexamples
Do not call a baseline fair when conditions are missing, metrics are incomparable, or leaderboard values are copied without protocol verification. Mark headroom unknown when the historical series is below its floor.
Provenance map
9 architecture old entries: baseline, inventory, extraction, standardization, discrepancy, progress, leaderboard, normalization, curve construction. Repeated reporting prose compressed.
Legacy context checkpoint / Delta notes
Append method IDs, normalized records, excluded records with reasons, discrepancy pairs, progress model, and headroom uncertainty.
Preserved source criteria ledger
| source |
source line |
kind |
source criterion |
| baseline-establishment |
28 |
textual |
\ |
| baseline-establishment |
40 |
textual |
\ |
| baseline-establishment |
58 |
textual |
\ |
| baseline-establishment |
70 |
numeric-table |
\ |
| baseline-establishment |
71 |
numeric-table |
\ |
| baseline-establishment |
72 |
numeric-table |
\ |
| baseline-establishment |
73 |
numeric-table |
\ |
| baseline-establishment |
74 |
numeric-table |
\ |
| baseline-establishment |
75 |
numeric-table |
\ |
| method-inventory |
23 |
numeric-table |
\ |
| method-inventory |
24 |
numeric-table |
\ |
| method-inventory |
25 |
numeric-table |
\ |
| method-inventory |
30 |
textual |
|
| method-inventory |
33 |
numeric-table |
\ |
| method-inventory |
34 |
numeric-table |
\ |
| method-inventory |
35 |
numeric-table |
\ |
| method-inventory |
36 |
numeric-table |
\ |
| method-inventory |
37 |
numeric-table |
\ |
| method-inventory |
38 |
textual |
|
| method-inventory |
41 |
numeric |
Cannot exit until methods_discovered >= 40 (80% of target). |
| performance-extraction |
18 |
textual |
Extract structured performance data from papers, leaderboards, and reproducibility studies. Each data point is a (Task, Dataset, Metric, Score, Conditions) tuple with full provenance. Prioritizes primary sources (original papers) but cross-references against leaderboards and third-party reproductions. |
| performance-extraction |
24 |
numeric-table |
\ |
| performance-extraction |
25 |
numeric-table |
\ |
| performance-extraction |
26 |
numeric-table |
\ |
| performance-extraction |
27 |
numeric-table |
\ |
| performance-extraction |
32 |
textual |
|
| performance-extraction |
35 |
numeric-table |
\ |
| performance-extraction |
36 |
numeric-table |
\ |
| performance-extraction |
37 |
numeric-table |
\ |
| performance-extraction |
38 |
numeric-table |
\ |
| performance-extraction |
39 |
numeric-table |
\ |
| performance-extraction |
40 |
numeric-table |
\ |
| performance-extraction |
41 |
textual |
|
| performance-extraction |
44 |
numeric |
Cannot exit until data_points >= 120 (80% of target). |
| condition-standardization |
25 |
numeric-table |
\ |
| condition-standardization |
26 |
numeric-table |
\ |
| condition-standardization |
27 |
numeric-table |
\ |
| condition-standardization |
28 |
numeric-table |
\ |
| condition-standardization |
33 |
textual |
|
| condition-standardization |
36 |
numeric-table |
\ |
| condition-standardization |
37 |
numeric-table |
\ |
| condition-standardization |
38 |
numeric-table |
\ |
| condition-standardization |
39 |
numeric-table |
\ |
| condition-standardization |
40 |
numeric-table |
\ |
| condition-standardization |
41 |
textual |
|
| condition-standardization |
44 |
numeric |
Cannot exit until data_points_standardized >= 48 (80% of target). |
| condition-standardization |
60 |
textual |
3. Group methods by comparable condition sets |
| condition-standardization |
63 |
textual |
6. Produce fair comparison subsets where conditions are controlled |
| discrepancy-analysis |
24 |
numeric-table |
\ |
| discrepancy-analysis |
25 |
numeric-table |
\ |
| discrepancy-analysis |
26 |
numeric-table |
\ |
| discrepancy-analysis |
27 |
numeric-table |
\ |
| discrepancy-analysis |
32 |
textual |
|
| discrepancy-analysis |
35 |
numeric-table |
\ |
| discrepancy-analysis |
36 |
numeric-table |
\ |
| discrepancy-analysis |
37 |
numeric-table |
\ |
| discrepancy-analysis |
38 |
numeric-table |
\ |
| discrepancy-analysis |
39 |
numeric-table |
\ |
| discrepancy-analysis |
40 |
textual |
|
| discrepancy-analysis |
43 |
numeric |
Cannot exit until score_pairs_compared >= 36 (80% of target). |
| discrepancy-analysis |
52 |
textual |
- reproducibility-checklist-audit - Assess paper reproducibility completeness |
| discrepancy-analysis |
59 |
textual |
4. Apply reproducibility-checklist-audit to papers with large discrepancies |
| discrepancy-analysis |
85 |
textual |
"reproducibility_checklist_score": 0, |
| progress-quantification |
26 |
numeric-table |
\ |
| progress-quantification |
27 |
numeric-table |
\ |
| progress-quantification |
28 |
numeric-table |
\ |
| progress-quantification |
29 |
numeric-table |
\ |
| progress-quantification |
34 |
textual |
|
| progress-quantification |
37 |
numeric-table |
\ |
| progress-quantification |
38 |
numeric-table |
\ |
| progress-quantification |
39 |
numeric-table |
\ |
| progress-quantification |
40 |
numeric-table |
\ |
| progress-quantification |
41 |
numeric-table |
\ |
| progress-quantification |
42 |
numeric-table |
\ |
| progress-quantification |
43 |
textual |
|
| progress-quantification |
46 |
numeric |
Cannot exit until historical_data_points >= 80 (80% of target). |
| leaderboard-harvesting |
43 |
numeric |
- Flag discrepancies > 1 standard deviation |
| leaderboard-harvesting |
58 |
textual |
## Minimum Yield |
| leaderboard-harvesting |
62 |
numeric-table |
\ |
| leaderboard-harvesting |
63 |
numeric-table |
\ |
| leaderboard-harvesting |
64 |
numeric-table |
\ |
| leaderboard-harvesting |
65 |
numeric-table |
\ |
| condition-normalization |
28 |
textual |
- Random seeds: number of runs, seed selection, variance reported |
| condition-normalization |
51 |
textual |
### Stage 4: Fair Comparison Baseline |
| condition-normalization |
53 |
textual |
Apply normalization to produce fair comparison subsets: |
| condition-normalization |
58 |
textual |
Yield: Fair comparison tables with methodology notes. |
| condition-normalization |
60 |
textual |
## Minimum Yield |
| condition-normalization |
64 |
numeric-table |
\ |
| condition-normalization |
65 |
numeric-table |
\ |
| condition-normalization |
66 |
numeric-table |
\ |
| condition-normalization |
67 |
numeric-table |
\ |
| progress-curve-construction |
62 |
textual |
## Minimum Yield |
| progress-curve-construction |
66 |
numeric-table |
\ |
| progress-curve-construction |
67 |
numeric-table |
\ |
| progress-curve-construction |
68 |
numeric-table |
\ |
| progress-curve-construction |
69 |
numeric-table |
\ |
| progress-curve-construction |
73 |
numeric |
- progress-curve-fitting (for Stages 2-3) |
Context checkpoint / Delta notes
Append method IDs, normalized records, excluded records with reasons, discrepancy pairs, progress model, and headroom uncertainty.
1---2name: establish-empirical-baseline3description: "Establish a fair empirical baseline by inventorying methods, extracting comparable performance, normalizing conditions/compute, checking discrepancies, and estimating progress/headroom."4metadata:5 source: https://skillmd.com/skills/yogsoth-ai/establish-empirical-baseline6---78# establish-empirical-baseline910## Purpose1112Establish a fair empirical baseline by inventorying methods, extracting comparable performance, normalizing conditions/compute, checking discrepancies, and estimating progress/headroom.1314## Input contract1516```yaml17required: [method_records, benchmark_or_task, performance_measure]18optional: [historical_series, compute_metadata, condition_schema]19constraints: [comparability fields and source provenance required]20```2122## Execution protocol2324Do not perform called SOP operations inline; each loaded SOP owns its contract and thresholds.25261. You MUST load skill `inventory-reference-items` to inventory methods and comparison records.272. You MUST load skill `extract-evidence-record` to extract performance evidence. You MUST load skill `audit-reporting-quality` to audit missing or ambiguous reporting. You MUST load skill `normalize-comparison-scale` to normalize units, data, compute, and evaluation protocol.283. You MUST load skill `detect-performance-discrepancy` to detect comparison discrepancies. You MUST load skill `estimate-performance-headroom` to estimate headroom. You MUST load skill `analyze-temporal-trajectory` to analyze progress and leaderboard dynamics. You MUST load skill `check-dominance` to identify dominated and incomparable records.294. Synthesize the baseline with uncertainty and known incomparable records.30 If the resulting methods or baselines require explicit comparative selection, consider `rank-candidates` as the next tactic.3132## Output contract3334```yaml35produces: [method_inventory, normalized_baseline, discrepancy_report, progress_curve, headroom_estimate]36delta_fields: [findings, evidence_updates, uncertainties, decisions, open_questions]37```3839## Thresholds and quality gates4041- Baseline acquisition gates are relative to a declared eligible universe: record numerator, denominator, batch increment, stopping reason, and source references for each ratio.42- `method-inventory`: method coverage ratio reaches a justified floor over the eligible method universe.43- `performance-extraction`: comparable-record coverage ratio reaches a justified floor over eligible records.44- `condition-standardization`: complete condition-vector ratio reaches a justified floor over comparable records.45- `discrepancy-analysis`: score-pair coverage ratio reaches a justified floor over eligible comparison pairs.46- `progress-quantification`: historical-time coverage and independent-source ratio reach justified floors; stop when added periods no longer change the trajectory conclusion.47- Across modes, report marginal information gain and saturation state when added records or periods no longer change the baseline conclusion.48- Normalization must expose condition, compute, metric, and unit transformations.4950## Failure and counterexamples5152Do not call a baseline fair when conditions are missing, metrics are incomparable, or leaderboard values are copied without protocol verification. Mark headroom unknown when the historical series is below its floor.5354## Provenance map55569 architecture `old` entries: baseline, inventory, extraction, standardization, discrepancy, progress, leaderboard, normalization, curve construction. Repeated reporting prose compressed.5758## Legacy context checkpoint / Delta notes5960Append method IDs, normalized records, excluded records with reasons, discrepancy pairs, progress model, and headroom uncertainty.6162## Preserved source criteria ledger6364| source | source line | kind | source criterion |65|---|---:|---|---|66| baseline-establishment | 28 | textual | \\| Check reproducibility / discrepancies \\| discrepancy-analysis \\| |67| baseline-establishment | 40 | textual | \\| discrepancy-analysis \\| Identify discrepancies between reported and reproducible scores \\| |68| baseline-establishment | 58 | textual | \\| reproducibility-checklist-audit \\| Assess paper against ML Reproducibility Checklist \\| |69| baseline-establishment | 70 | numeric-table | \\| method-inventory \\| 50 \\| 0 \\| 60 \\| |70| baseline-establishment | 71 | numeric-table | \\| performance-extraction \\| 30 \\| 150 \\| 40 \\| |71| baseline-establishment | 72 | numeric-table | \\| condition-standardization \\| 20 \\| 60 \\| 30 \\| |72| baseline-establishment | 73 | numeric-table | \\| discrepancy-analysis \\| 15 \\| 45 \\| 30 \\| |73| baseline-establishment | 74 | numeric-table | \\| progress-quantification \\| 30 \\| 100 \\| 40 \\| |74| baseline-establishment | 75 | numeric-table | \\| **TOTAL** \\| **145** \\| **355** \\| **200** \\| |75| method-inventory | 23 | numeric-table | \\| Methods discovered \\| 30 \\| 50 \\| |76| method-inventory | 24 | numeric-table | \\| Web searches \\| 40 \\| 60 \\| |77| method-inventory | 25 | numeric-table | \\| Papers consulted \\| 20 \\| 40 \\| |78| method-inventory | 30 | textual | <HARD-GATE> |79| method-inventory | 33 | numeric-table | \\| Methods discovered \\| 0 \\| 50 \\| BLOCKED \\| |80| method-inventory | 34 | numeric-table | \\| Web searches used \\| 0 \\| 60 \\| - \\| |81| method-inventory | 35 | numeric-table | \\| Papers consulted \\| 0 \\| 40 \\| - \\| |82| method-inventory | 36 | numeric-table | \\| Leaderboard sources \\| 0 \\| 5 \\| - \\| |83| method-inventory | 37 | numeric-table | \\| Citation chains traced \\| 0 \\| 10 \\| - \\| |84| method-inventory | 38 | textual | </HARD-GATE> |85| method-inventory | 41 | numeric | Cannot exit until methods_discovered >= 40 (80% of target). |86| performance-extraction | 18 | textual | Extract structured performance data from papers, leaderboards, and reproducibility studies. Each data point is a (Task, Dataset, Metric, Score, Conditions) tuple with full provenance. Prioritizes primary sources (original papers) but cross-references against leaderboards and third-party reproductions. |87| performance-extraction | 24 | numeric-table | \\| Methods covered \\| 20 \\| 30 \\| |88| performance-extraction | 25 | numeric-table | \\| Data points extracted \\| 100 \\| 150 \\| |89| performance-extraction | 26 | numeric-table | \\| Web searches \\| 25 \\| 40 \\| |90| performance-extraction | 27 | numeric-table | \\| Papers read \\| 15 \\| 30 \\| |91| performance-extraction | 32 | textual | <HARD-GATE> |92| performance-extraction | 35 | numeric-table | \\| Methods covered \\| 0 \\| 30 \\| BLOCKED \\| |93| performance-extraction | 36 | numeric-table | \\| Data points extracted \\| 0 \\| 150 \\| BLOCKED \\| |94| performance-extraction | 37 | numeric-table | \\| Web searches used \\| 0 \\| 40 \\| - \\| |95| performance-extraction | 38 | numeric-table | \\| Papers read \\| 0 \\| 30 \\| - \\| |96| performance-extraction | 39 | numeric-table | \\| Datasets covered \\| 0 \\| 5 \\| - \\| |97| performance-extraction | 40 | numeric-table | \\| Metrics tracked \\| 0 \\| 3 \\| - \\| |98| performance-extraction | 41 | textual | </HARD-GATE> |99| performance-extraction | 44 | numeric | Cannot exit until data_points >= 120 (80% of target). |100| condition-standardization | 25 | numeric-table | \\| Methods analyzed \\| 15 \\| 20 \\| |101| condition-standardization | 26 | numeric-table | \\| Data points standardized \\| 40 \\| 60 \\| |102| condition-standardization | 27 | numeric-table | \\| Web searches \\| 20 \\| 30 \\| |103| condition-standardization | 28 | numeric-table | \\| Condition dimensions cataloged \\| 5 \\| 10 \\| |104| condition-standardization | 33 | textual | <HARD-GATE> |105| condition-standardization | 36 | numeric-table | \\| Methods analyzed \\| 0 \\| 20 \\| BLOCKED \\| |106| condition-standardization | 37 | numeric-table | \\| Data points standardized \\| 0 \\| 60 \\| BLOCKED \\| |107| condition-standardization | 38 | numeric-table | \\| Condition dimensions \\| 0 \\| 10 \\| - \\| |108| condition-standardization | 39 | numeric-table | \\| Normalization rules defined \\| 0 \\| 5 \\| - \\| |109| condition-standardization | 40 | numeric-table | \\| Fair comparison sets \\| 0 \\| 3 \\| - \\| |110| condition-standardization | 41 | textual | </HARD-GATE> |111| condition-standardization | 44 | numeric | Cannot exit until data_points_standardized >= 48 (80% of target). |112| condition-standardization | 60 | textual | 3. Group methods by comparable condition sets |113| condition-standardization | 63 | textual | 6. Produce fair comparison subsets where conditions are controlled |114| discrepancy-analysis | 24 | numeric-table | \\| Methods analyzed \\| 10 \\| 15 \\| |115| discrepancy-analysis | 25 | numeric-table | \\| Data points compared \\| 30 \\| 45 \\| |116| discrepancy-analysis | 26 | numeric-table | \\| Web searches \\| 20 \\| 30 \\| |117| discrepancy-analysis | 27 | numeric-table | \\| Reproduction studies consulted \\| 5 \\| 10 \\| |118| discrepancy-analysis | 32 | textual | <HARD-GATE> |119| discrepancy-analysis | 35 | numeric-table | \\| Methods analyzed \\| 0 \\| 15 \\| BLOCKED \\| |120| discrepancy-analysis | 36 | numeric-table | \\| Score pairs compared \\| 0 \\| 45 \\| BLOCKED \\| |121| discrepancy-analysis | 37 | numeric-table | \\| Discrepancies flagged \\| 0 \\| - \\| - \\| |122| discrepancy-analysis | 38 | numeric-table | \\| Reproduction studies found \\| 0 \\| 10 \\| - \\| |123| discrepancy-analysis | 39 | numeric-table | \\| Reliability ratings assigned \\| 0 \\| 15 \\| - \\| |124| discrepancy-analysis | 40 | textual | </HARD-GATE> |125| discrepancy-analysis | 43 | numeric | Cannot exit until score_pairs_compared >= 36 (80% of target). |126| discrepancy-analysis | 52 | textual | - **reproducibility-checklist-audit** - Assess paper reproducibility completeness |127| discrepancy-analysis | 59 | textual | 4. Apply reproducibility-checklist-audit to papers with large discrepancies |128| discrepancy-analysis | 85 | textual | "reproducibility_checklist_score": 0, |129| progress-quantification | 26 | numeric-table | \\| Methods tracked \\| 20 \\| 30 \\| |130| progress-quantification | 27 | numeric-table | \\| Historical data points \\| 70 \\| 100 \\| |131| progress-quantification | 28 | numeric-table | \\| Web searches \\| 25 \\| 40 \\| |132| progress-quantification | 29 | numeric-table | \\| Time span covered (years) \\| 3 \\| 5+ \\| |133| progress-quantification | 34 | textual | <HARD-GATE> |134| progress-quantification | 37 | numeric-table | \\| Methods tracked \\| 0 \\| 30 \\| BLOCKED \\| |135| progress-quantification | 38 | numeric-table | \\| Historical data points \\| 0 \\| 100 \\| BLOCKED \\| |136| progress-quantification | 39 | numeric-table | \\| Web searches used \\| 0 \\| 40 \\| - \\| |137| progress-quantification | 40 | numeric-table | \\| Progress curves built \\| 0 \\| 3 \\| - \\| |138| progress-quantification | 41 | numeric-table | \\| Headroom estimates \\| 0 \\| 3 \\| - \\| |139| progress-quantification | 42 | numeric-table | \\| Inflection points identified \\| 0 \\| 2 \\| - \\| |140| progress-quantification | 43 | textual | </HARD-GATE> |141| progress-quantification | 46 | numeric | Cannot exit until historical_data_points >= 80 (80% of target). |142| leaderboard-harvesting | 43 | numeric | - Flag discrepancies > 1 standard deviation |143| leaderboard-harvesting | 58 | textual | ## Minimum Yield |144| leaderboard-harvesting | 62 | numeric-table | \\| Leaderboard sources checked \\| 3 \\| |145| leaderboard-harvesting | 63 | numeric-table | \\| Methods with scores \\| 15 \\| |146| leaderboard-harvesting | 64 | numeric-table | \\| Cross-validated score pairs \\| 10 \\| |147| leaderboard-harvesting | 65 | numeric-table | \\| Deduplication conflicts resolved \\| 5 \\| |148| condition-normalization | 28 | textual | - Random seeds: number of runs, seed selection, variance reported |149| condition-normalization | 51 | textual | ### Stage 4: Fair Comparison Baseline |150| condition-normalization | 53 | textual | Apply normalization to produce fair comparison subsets: |151| condition-normalization | 58 | textual | **Yield**: Fair comparison tables with methodology notes. |152| condition-normalization | 60 | textual | ## Minimum Yield |153| condition-normalization | 64 | numeric-table | \\| Condition dimensions cataloged \\| 5 \\| |154| condition-normalization | 65 | numeric-table | \\| Methods with full condition vectors \\| 10 \\| |155| condition-normalization | 66 | numeric-table | \\| Normalization rules defined \\| 3 \\| |156| condition-normalization | 67 | numeric-table | \\| Fair comparison sets produced \\| 2 \\| |157| progress-curve-construction | 62 | textual | ## Minimum Yield |158| progress-curve-construction | 66 | numeric-table | \\| Progress curves constructed \\| 2 \\| |159| progress-curve-construction | 67 | numeric-table | \\| Years of history covered \\| 3 \\| |160| progress-curve-construction | 68 | numeric-table | \\| Inflection points identified \\| 1 \\| |161| progress-curve-construction | 69 | numeric-table | \\| Headroom estimates produced \\| 2 \\| |162| progress-curve-construction | 73 | numeric | - progress-curve-fitting (for Stages 2-3) |163164## Context checkpoint / Delta notes165166Append method IDs, normalized records, excluded records with reasons, discrepancy pairs, progress model, and headroom uncertainty.