Experiment Design

Experiment Design

Interview-driven design workflow producing design documents, power analysis scripts, and pre-analysis plans.

Modes

Default: Design. If user provides a .qsf file, auto-select Survey mode.

When to Use

• Designing a new experiment or survey
• Calculating required sample sizes
• Writing or auditing a pre-analysis plan
• Parsing a Qualtrics .qsf file to understand its structure
• Building a survey specification from a natural language description

When NOT to Use

• Running the analysis → data-analysis
• Auditing identification strategy for observational studies → causal-design
• Generating synthetic test data → synthetic-data

Shared References

• Method probing questions: shared/method-probing-questions.md — ask before designing (Experiments/RCTs, Survey sections)
• Validation tiers: shared/validation-tiers.md — tier determines required power and pre-registration
• Escalation protocol: shared/escalation-protocol.md — escalate when design has validity threats
• Engagement-stratified sampling: shared/engagement-stratified-sampling.md — stratify social media samples by engagement
• Inter-coder reliability: shared/intercoder-reliability.md — reliability planning for content analysis designs

Mode: Power

Read references/power-analysis-recipes.md for language-specific code patterns.

Workflow

1. Interview — ask for:
   • Primary outcome variable and expected effect size (or domain norms)
   • Design type (between-subjects, within-subjects, factorial, cluster-randomized)
   • Number of conditions/groups
   • Significance level (default: 0.05) and desired power (default: 0.80)
   • Any clustering or stratification
2. Generate script — R (DeclareDesign/pwr) or Python (statsmodels.stats.power)
3. Execute and report — produce a sample size table showing N for power = {0.80, 0.90, 0.95}
4. Write to project — save script to code/power_analysis.R (or .py), results to output/power_analysis_results.md

HPC escalation: If the power analysis uses Monte Carlo simulation (e.g., DeclareDesign with >10k replications, or a multi-design sweep), move execution to [HPC cluster] — drop the simulation script into hpc/ with templates/slurm/array.sbatch (array over seeds/designs). The SHA-logging snippet in the template pins results to the DGP version. See docs/guides/hpc.md.

Effect Size Guidance

If the user doesn't know the expected effect size, guide them:

Never default to Cohen's benchmarks without acknowledging they are arbitrary.

Mode: Design

Workflow

1. Research question interview — structured questions:
   • What is the causal question?
   • What is the treatment / intervention?
   • What is the primary outcome? Secondary outcomes?
   • What is the target population?
   • What is the assignment mechanism? (random, stratified, clustered, matched)
2. Design specification — produce a structured document covering:
   • Hypotheses (directional, with expected signs)
   • Conditions (treatment arms, control)
   • Randomization procedure
   • Outcome measures and scales
   • Sample and recruitment strategy
   • Timeline
3. Identification check — state the estimand, identifying assumptions, and potential threats
4. Write design document — save to docs/experiment-design.md or project-appropriate location
5. Lock the design — record in MEMORY.md Estimand Registry (if project has one) or flag for the user to lock before analysis

This design document is what data-analysis Phase 3 checks for before allowing estimation.

Mode: PAP

Read references/pap-template.md for the full template structure.

Workflow

1. Check for existing design — look for design document in docs/, log/plans/, or .context/
2. If no design exists — run Design mode first, then continue
3. Generate PAP — structured pre-analysis plan following AEA/OSF/EGAP conventions:
   • Study information (title, authors, IRB, registration)
   • Design overview (hypotheses, conditions, randomization, sample)
   • Data collection (instruments, timing, blinding)
   • Analysis plan (main specification, inference, multiple testing corrections)
   • Robustness and sensitivity analyses
   • Power analysis (reference or embed results from Power mode)
4. Output — save to docs/pre-analysis-plan.md (or .tex if user prefers LaTeX)
5. Registration prompt — remind user to register at AEA RCT Registry, OSF, or EGAP

Mode: Survey

Survey mode is a first-class capability with two entry points: QSF parsing and natural language construction.

Entry Point A: QSF Parsing

When user provides a Qualtrics .qsf file:

1. Parse the JSON — extract survey flow, blocks, questions, embedded data, skip logic
2. Map question types — read references/qsf-parsing-guide.md for type mapping:
   • Matrix, Likert, slider, numeric, constant sum, rank order, best-worst scaling
   • Text entry (single-line, multi-line, essay)
   • Multiple choice (single answer, multiple answer)
3. Detect design elements:
   • Factorial conditions from randomizer blocks and embedded data
   • Attention checks and comprehension checks
   • Known validated scales (read references/known-scales-registry.md)
   • Skip/display logic and branching
4. Produce structured summary — questions, conditions, scales, logic flow, warnings
5. Flag issues — missing attention checks, unbalanced conditions, potential order effects

Entry Point B: Natural Language Construction

When user describes an experiment in natural language:

1. Parse the design — extract factorial structure from text
   • Example: "3 (Source: AI vs Human vs None) x 2 (Product: Hedonic vs Utilitarian)" → 3x2 between-subjects
2. Interview for details:
   • What DVs to measure? (recommend appropriate scale types)
   • What manipulation checks?
   • What attention/comprehension checks?
   • Demographics and control variables?
3. Generate survey specification — structured document with:
   • Survey flow (consent → demographics → manipulation → DVs → manipulation check → debrief)
   • Question text and response options for each item
   • Condition assignments and randomization logic
   • Attention check placement (read references/survey-design-checklist.md)
4. Scale recommendations — read references/known-scales-registry.md for validated scales matching the constructs

Survey Quality Checks

Read references/survey-design-checklist.md for the full checklist. Key checks:

• Attention checks: At least 1 per 5 minutes of survey length. Calibrated pass rates (85-95%). See Krosnick (1991), Meade & Craig (2012).
• Response style mitigation: Flag scales vulnerable to acquiescence bias. Recommend reverse-coding for scales with 4+ items in the same direction.
• Scale quality: Semantic type detection (satisfaction, trust, intention, risk). Reverse-coded item flagging. Well-known scale recognition.
• Order effects: Randomize question order within blocks where appropriate. Flag potential priming from early questions.

Cross-References