yogsoth-ai Tactic: Statistical Method Selection
Orchestration Pattern
- Assess Data Characteristics → Determine distribution type, sample size, pairing structure
- metric-specification → Ensure metrics are well-defined and measurable
- Select Test Family → Choose between parametric, non-parametric, or Bayesian
- sample-size-estimation → Power analysis for the selected test
- Define Analysis Pipeline → Pre-register the complete analysis plan
Decision Criteria
| Condition | Recommended Method |
|---|---|
| Normal data, 2 groups, paired | Paired t-test |
| Normal data, 2 groups, unpaired | Welch's t-test |
| Normal data, 3+ groups | ANOVA + post-hoc (Tukey HSD) |
| Non-normal, 2 groups | Wilcoxon signed-rank / Mann-Whitney U |
| Non-normal, 3+ groups | Kruskal-Wallis + Dunn's test |
| Multiple datasets, multiple methods | Friedman + Nemenyi / critical difference |
| Want probability of superiority | Bayesian comparison (Benavoli 2017) |
| Small sample, no distributional assumptions | Permutation test |
| Variance estimation needed | Bootstrap confidence intervals |
| Multiple comparisons | Apply Holm-Bonferroni correction |
Quality Checks
- Is the test appropriate for the data type (continuous, ordinal, categorical)?
- Are independence assumptions met? (If not, use paired/repeated-measures variants)
- Is the sample size sufficient for the chosen test's power requirements?
- Are multiple comparison corrections applied when testing multiple hypotheses?
- Is the effect size reported alongside p-values?
- Is the significance threshold pre-registered (not chosen post-hoc)?
Available SOPs
Optional, no fixed order; the final leaf is always a sop.
| SOP | When to use |
|---|---|
| metric-specification | Define experiment metrics and significance standards |
| sample-size-estimation | SOP: power analysis and required experiment count estimation |