Design of Experiments — Diagnostics & Primary Analysis
Table of Contents
Open-source skill.
Scope Boundary
Use this skill when:
- The data come from a designed experiment with controlled factors and a clear error structure.
- A baseline factorial ANOVA-style analysis is the right first check before optimization or alias-structure work.
Do not use this skill when:
- The study is observational rather than experimental.
- The main need is repeated-measures, longitudinal, or SEM modeling rather than DOE analysis.
Workflow
Read each step file in workflow/ before executing that step.
| Step |
Responsibility |
Executor |
Document |
Input |
Output |
| Collect |
Collect Inputs |
Main Agent |
workflow/step01-collect-inputs.md |
User input |
Structured input summary |
| Diagnose |
Check Distribution |
Main Agent |
workflow/step02-check-distribution.md |
Prior step output |
PART 1 code block |
| Test |
Run Primary Test |
Main Agent |
workflow/step03-run-primary-test.md |
Prior step output |
PART 2-3 code blocks |
Decision Tree
1. CHECK DESIGN
├── Balanced (equal N per cell) → standard ANOVA
└── Unbalanced → Type III SS mandatory
2. CHECK RESIDUALS
├── Normal residuals + equal variances → factorial ANOVA
└── Non-normal / heteroscedastic → flag, proceed with caution + recommend transformations
3. FACTORIAL ANOVA
├── Significant main effects → Tukey HSD post-hoc
└── Significant interactions → simple effects analysis
Required Inputs
| Role |
What to collect |
| Response (Y) |
Variable name, units, what it measures |
| Factors |
Names, levels, type (fixed/random) |
| Design type |
CRD, RCBD, factorial, fractional, split-plot, Latin square — if split-plot, identify the whole-plot factor and sub-plot factor explicitly, because the design requires two separate error terms (whole-plot error and sub-plot error); analyzing them against a single pooled error inflates Type I error on the whole-plot factor and deflates it on the sub-plot factor. Generated code MUST specify Error(whole_plot_id / sub_plot_factor) in R's aov() or `random = ~1 |
| Blocking variable |
If RCBD or similar |
| Replication info |
Replicates per cell, total N |
Code Structure
PART 0: Setup & Data Loading
PART 1: Design Diagnostics → plot_01_interaction.png
PART 2: Factorial ANOVA + Post-hoc → plot_02_effects.png
PART 3: Recommendation Block → text listing additional analyses available
Reporting Standards
- p-values: "< .001" not "0.000"; exact to 3 decimals otherwise
- Effect sizes: partial eta-squared always alongside F-tests
- SS Type III always for unbalanced designs
- F(df1, df2) = X.XX, p, partial eta-squared
- Effect estimates with SE
- Degrees of freedom: always with F statistics
- Sample size: final analytic N and cell sizes
- Decimal places: 2 for M/SD, 3 for p and effect sizes
- Non-significance: "not statistically significant at alpha = .05" -- never "no effect"
- Design resolution for fractional factorial (in recommendation)
Hypothesis Tests
| Scenario |
Method |
| Main effects |
F-test from factorial ANOVA |
| Interactions |
F-test for each interaction term |
| Post-hoc (main effects) |
Tukey HSD (all pairwise) |
| Post-hoc (interactions) |
Simple effects (effect of A at each level of B) |
| Variance homogeneity |
Levene's test |
| Residual normality |
Shapiro-Wilk on residuals |
Example Dataset
R built-in npk: 24 observations, N/P/K fertilizer treatments (2^3 factorial)
on pea yield, blocked by block. Classic agricultural DOE.
Python: sm.datasets.get_rdataset("npk").data (with offline fallback to bundled examples/npk.csv).
Alternative: warpbreaks -- wool x tension factorial on breaks.
Method Status
| Status |
Methods |
| Implemented in this skill |
Design-balance checks, residual diagnostics, factorial ANOVA, Tukey HSD, and simple-effects escalation triggers |
Implemented downstream in vera-data-doe-generating |
RSM, fractional-factorial alias analysis, split-plot handling, desirability optimization, and exploratory trees |
| Out of scope in this open-source baseline |
Observational causal modeling and experimental workflows not grounded in a named DOE design |
Minimal Smoke Test
- Smoke-test prompt: "Run
vera-data-doe-reviewing on npk, treating yield as the response and N, P, and K as the factorial treatments. Produce the standard baseline artifacts."
Cross-Skill Interface
Output:
├── code_r → .R script
├── code_python → .py script
├── figures/ → 2 PNGs (interaction plot + effects plot)
└── recommendations → text block (additional analyses available)
Next step: Invoke vera-data-doe-generating from this skillset to run the full pipeline (additional tests, subgroup analysis, modeling, manuscript generation). See ../../CROSS-SKILL-INTERFACE.md for the shared handoff contract.
1---2name: vera-data-doe-reviewing3description: Runs distribution diagnostics and primary analysis for designed experiments (DOE) with controlled factors. Produces cell descriptives, balance check, residual normality, Levene's test, full factorial ANOVA (Type III SS) with partial eta-squared, interaction plots, and Tukey HSD / simple effects post-hoc. Outputs .R and .py scripts with 2 publication-quality plots. Triggered when user mentions "experimental design," "DOE," "factorial design," "treatment effects," "blocking," "randomized experiment," "split-plot," "response surface," "2^k factorial," "main effects and interactions," "Latin square," "fractional factorial," "CRD," "RCBD," or describes manipulated factors. Fractional factorial and RSM are identified and flagged; full design-resolution analysis delegated to vera-data-doe-generating. Does not handle observational studies, repeated measures, or SEM outcomes.4---56# Design of Experiments — Diagnostics & Primary Analysis78## Table of Contents910- [Scope Boundary](#scope-boundary)11- [Workflow](#workflow)12- [Decision Tree](#decision-tree)13- [Required Inputs](#required-inputs)14- [Code Structure](#code-structure)15- [Reporting Standards](#reporting-standards)16- [Hypothesis Tests](#hypothesis-tests)17- [Example Dataset](#example-dataset)18- [Method Status](#method-status)19- [Minimal Smoke Test](#minimal-smoke-test)20- [Cross-Skill Interface](#cross-skill-interface)212223Open-source skill.2425## Scope Boundary2627Use this skill when:28- The data come from a designed experiment with controlled factors and a clear error structure.29- A baseline factorial ANOVA-style analysis is the right first check before optimization or alias-structure work.3031Do not use this skill when:32- The study is observational rather than experimental.33- The main need is repeated-measures, longitudinal, or SEM modeling rather than DOE analysis.3435## Workflow3637Read each step file in `workflow/` before executing that step.3839| Step | Responsibility | Executor | Document | Input | Output |40|---|---|---|---|---|---|41| Collect | Collect Inputs | Main Agent | `workflow/step01-collect-inputs.md` | User input | Structured input summary |42| Diagnose | Check Distribution | Main Agent | `workflow/step02-check-distribution.md` | Prior step output | PART 1 code block |43| Test | Run Primary Test | Main Agent | `workflow/step03-run-primary-test.md` | Prior step output | PART 2-3 code blocks |4445## Decision Tree4647```481. CHECK DESIGN49 ├── Balanced (equal N per cell) → standard ANOVA50 └── Unbalanced → Type III SS mandatory51522. CHECK RESIDUALS53 ├── Normal residuals + equal variances → factorial ANOVA54 └── Non-normal / heteroscedastic → flag, proceed with caution + recommend transformations55563. FACTORIAL ANOVA57 ├── Significant main effects → Tukey HSD post-hoc58 └── Significant interactions → simple effects analysis59```6061## Required Inputs6263| Role | What to collect |64|---|---|65| **Response (Y)** | Variable name, units, what it measures |66| **Factors** | Names, levels, type (fixed/random) |67| **Design type** | CRD, RCBD, factorial, fractional, **split-plot**, Latin square — if split-plot, identify the whole-plot factor and sub-plot factor explicitly, because the design requires **two separate error terms** (whole-plot error and sub-plot error); analyzing them against a single pooled error inflates Type I error on the whole-plot factor and deflates it on the sub-plot factor. Generated code MUST specify `Error(whole_plot_id / sub_plot_factor)` in R's `aov()` or `random = ~1 | whole_plot_id` in `lme()`. |68| **Blocking variable** | If RCBD or similar |69| **Replication info** | Replicates per cell, total N |7071## Code Structure7273```74PART 0: Setup & Data Loading75PART 1: Design Diagnostics → plot_01_interaction.png76PART 2: Factorial ANOVA + Post-hoc → plot_02_effects.png77PART 3: Recommendation Block → text listing additional analyses available78```7980## Reporting Standards81821. p-values: "< .001" not "0.000"; exact to 3 decimals otherwise832. Effect sizes: partial eta-squared always alongside F-tests843. SS Type III always for unbalanced designs854. F(df1, df2) = X.XX, p, partial eta-squared865. Effect estimates with SE876. Degrees of freedom: always with F statistics887. Sample size: final analytic N and cell sizes898. Decimal places: 2 for M/SD, 3 for p and effect sizes909. Non-significance: "not statistically significant at alpha = .05" -- never "no effect"9110. Design resolution for fractional factorial (in recommendation)9293## Hypothesis Tests9495| Scenario | Method |96|---|---|97| Main effects | F-test from factorial ANOVA |98| Interactions | F-test for each interaction term |99| Post-hoc (main effects) | Tukey HSD (all pairwise) |100| Post-hoc (interactions) | Simple effects (effect of A at each level of B) |101| Variance homogeneity | Levene's test |102| Residual normality | Shapiro-Wilk on residuals |103104## Example Dataset105106R built-in `npk`: 24 observations, N/P/K fertilizer treatments (2^3 factorial)107on pea yield, blocked by block. Classic agricultural DOE.108Python: `sm.datasets.get_rdataset("npk").data` (with offline fallback to bundled `examples/npk.csv`).109110Alternative: `warpbreaks` -- wool x tension factorial on breaks.111112## Method Status113114| Status | Methods |115|---|---|116| Implemented in this skill | Design-balance checks, residual diagnostics, factorial ANOVA, Tukey HSD, and simple-effects escalation triggers |117| Implemented downstream in `vera-data-doe-generating` | RSM, fractional-factorial alias analysis, split-plot handling, desirability optimization, and exploratory trees |118| Out of scope in this open-source baseline | Observational causal modeling and experimental workflows not grounded in a named DOE design |119120## Minimal Smoke Test121122- Smoke-test prompt: "Run `vera-data-doe-reviewing` on `npk`, treating `yield` as the response and `N`, `P`, and `K` as the factorial treatments. Produce the standard baseline artifacts."123124## Cross-Skill Interface125126```127Output:128├── code_r → .R script129├── code_python → .py script130├── figures/ → 2 PNGs (interaction plot + effects plot)131└── recommendations → text block (additional analyses available)132```133134Next step: Invoke `vera-data-doe-generating` from this skillset to run the full pipeline (additional tests, subgroup analysis, modeling, manuscript generation). See `../../CROSS-SKILL-INTERFACE.md` for the shared handoff contract.
VeraSuperHub