Strategy: Result Analysis
Key Question: What do the results tell us?
Methodology
Three-layer analysis combining frequentist, resampling, and Bayesian approaches:
- Statistical Testing — Bootstrap CI, Permutation tests, Bayesian ROPE judgment
- Effect Size Calculation — Cohen's d, Cliff's delta, or domain-appropriate measure
- Reproducibility Verification — Re-run with different seeds, compare distributions
- Synthesis — Integrate findings into actionable conclusions
Execution Flow
[Collected results from experiment-running]
→ statistical-testing (bootstrap/permutation/Bayesian)
→ effect size calculation
→ reproducibility-verification (re-run, compare)
→ execution-synthesis (comprehensive report)
→ OUTPUT: validated findings with confidence levels
Budget Gate
| Step |
Max Budget |
Output |
| Statistical testing |
8% |
Test results with p-values/CIs |
| Reproducibility |
8% |
Re-run comparison |
| Synthesis |
4% |
Final report |
Key Decisions
- Test selection:
- Known distribution → parametric (t-test, ANOVA)
- Unknown/non-normal → bootstrap CI or permutation test
- Need practical significance → Bayesian ROPE
- Reproducibility threshold: Results must agree within 1 SE across re-runs
- Effect size interpretation:
- Small: d < 0.2 (may not be practically significant)
- Medium: 0.2 ≤ d < 0.8 (likely meaningful)
- Large: d ≥ 0.8 (strong effect)
- ROPE (Region of Practical Equivalence): Define before testing, not after
Integration with Knowledge System
Results feed back into:
- Wiki vault (claims with evidence)
- Future experiment design (what worked, what didn't)
- North star progress tracking
Available Tactics
Optional, no fixed order; the final leaf is always a sop.
| Tactic |
When to use |
| result-validation-loop |
Validate results through statistical testing, ROPE judgment, reproducibility re-runs, and final synthesis |
Available SOPs
Optional, no fixed order; the final leaf is always a sop.
| SOP |
When to use |
| execution-synthesis |
Synthesize complete execution report from all results, tests, and reproducibility data |
| reproducibility-verification |
Verify result reproducibility via re-runs with different seeds and ICC comparison |
| statistical-testing |
Execute statistical tests — bootstrap, permutation, Bayesian ROPE — on experiment results |
1---2name: result-analysis3description: Statistically analyze collected results, verify reproducibility, and synthesize findings4---56# Strategy: Result Analysis78**Key Question**: What do the results tell us?910## Methodology1112Three-layer analysis combining frequentist, resampling, and Bayesian approaches:13141. **Statistical Testing** — Bootstrap CI, Permutation tests, Bayesian ROPE judgment152. **Effect Size Calculation** — Cohen's d, Cliff's delta, or domain-appropriate measure163. **Reproducibility Verification** — Re-run with different seeds, compare distributions174. **Synthesis** — Integrate findings into actionable conclusions1819## Execution Flow2021```22[Collected results from experiment-running]23 → statistical-testing (bootstrap/permutation/Bayesian)24 → effect size calculation25 → reproducibility-verification (re-run, compare)26 → execution-synthesis (comprehensive report)27 → OUTPUT: validated findings with confidence levels28```2930## Budget Gate3132| Step | Max Budget | Output |33|------|-----------|--------|34| Statistical testing | 8% | Test results with p-values/CIs |35| Reproducibility | 8% | Re-run comparison |36| Synthesis | 4% | Final report |3738## Key Decisions3940- **Test selection**: 41 - Known distribution → parametric (t-test, ANOVA)42 - Unknown/non-normal → bootstrap CI or permutation test43 - Need practical significance → Bayesian ROPE44- **Reproducibility threshold**: Results must agree within 1 SE across re-runs45- **Effect size interpretation**:46 - Small: d < 0.2 (may not be practically significant)47 - Medium: 0.2 ≤ d < 0.8 (likely meaningful)48 - Large: d ≥ 0.8 (strong effect)49- **ROPE (Region of Practical Equivalence)**: Define before testing, not after5051## Integration with Knowledge System5253Results feed back into:54- Wiki vault (claims with evidence)55- Future experiment design (what worked, what didn't)56- North star progress tracking5758<!-- BEGIN available-tables (generated) -->5960## Available Tactics6162Optional, no fixed order; the final leaf is always a sop.6364| Tactic | When to use |65| --- | --- |66| result-validation-loop | Validate results through statistical testing, ROPE judgment, reproducibility re-runs, and final synthesis |6768## Available SOPs6970Optional, no fixed order; the final leaf is always a sop.7172| SOP | When to use |73| --- | --- |74| execution-synthesis | Synthesize complete execution report from all results, tests, and reproducibility data |75| reproducibility-verification | Verify result reproducibility via re-runs with different seeds and ICC comparison |76| statistical-testing | Execute statistical tests — bootstrap, permutation, Bayesian ROPE — on experiment results |7778<!-- END available-tables (generated) -->
yogsoth-ai