qa-aman Overview
Based on Research Design (Creswell & Creswell, 6th ed.) and Research Methods in Education (Cohen, Manion & Morrison, 8th ed.). A study design is the architectural plan for a research project: it specifies how data will be collected, from whom, by what method, and how conclusions will be drawn. Creswell's rule: the research question determines the method. Do not choose a method first and then fit the question to it.
The test: given only this study design document, could a different researcher conduct the same study and expect to produce comparable results?
Workflow
Step 1: Choose the research paradigm
Select based on what the research question is asking:
| Question type | Paradigm | Example |
|---|---|---|
| "How much / how many / does X cause Y?" | Quantitative | "Does intervention X reduce anxiety scores?" |
| "How / why / what is the experience of?" | Qualitative | "How do nurses experience burnout?" |
| "What is the scope AND the depth of?" | Mixed methods | "How prevalent is X, and why does it occur?" |
Quantitative: produces numbers, statistical significance, generalizable findings Qualitative: produces meaning, interpretation, transferable insights Mixed: both, in sequence or concurrently
Step 2: Select the research design
Quantitative designs:
- RCT (randomized controlled trial): strongest causal inference, participants randomly assigned to conditions
- Quasi-experimental: manipulation without randomization, useful when randomization is impractical
- Survey/correlational: measures relationships without manipulation
- Longitudinal: same participants measured over time
- Cross-sectional: one-time measurement across groups
Qualitative designs:
- Phenomenology: lived experience of a phenomenon
- Grounded theory: generate theory from data (no pre-existing theory)
- Case study: deep examination of a bounded case
- Ethnography: cultural practices through immersion
- Narrative inquiry: stories as primary data
Choose based on:
- Can you manipulate the IV? → experimental / quasi-experimental
- Are you exploring or confirming? → qualitative vs. quantitative
- Do you need causal claims or descriptive claims?
Step 3: Define the sample
Population: [full group you want findings to apply to]
Sampling strategy: [how you will select participants]
Quantitative options: random sampling, stratified random, cluster
Qualitative options: purposive, snowball, maximum variation, theoretical
Inclusion criteria: [who qualifies to participate]
Exclusion criteria: [who is excluded and why]
Sample size:
Quantitative: calculated via power analysis
Effect size (Cohen's d or f): [estimated from prior literature]
Power (1-β): 0.80 (standard) or 0.90 (preferred)
Alpha (α): 0.05
Required N per group: [result from power calculation]
Planned N: [required N + 15-20% buffer for attrition]
Qualitative: determined by saturation
Starting N: [e.g. 8-12 for phenomenology, 20-30 for grounded theory]
Saturation criterion: [stop when 3 consecutive interviews add no new themes]
Recruitment: [how will you find participants?]
Incentive: [compensation, if any]
Step 4: Document the procedure
Write the study procedure step-by-step, as a protocol:
Timeline: [total duration of study - e.g. 12 weeks]
Pre-study:
1. [IRB/ethics approval - required before data collection]
2. [Participant recruitment and screening]
3. [Informed consent process]
4. [Baseline measures collected]
Intervention/treatment (if applicable):
- Group A (experimental): [exact description of what participants receive]
- Group B (control): [exact description - active control or waitlist?]
- Duration: [N sessions / N weeks]
- Fidelity check: [how you verify the intervention was delivered as planned]
Data collection:
- Time points: [T1 = baseline, T2 = post-intervention, T3 = 3-month follow-up]
- Method: [survey / interview / observation / physiological measure / etc.]
- Instrument: [name of validated scale or instrument]
- Administration: [online / in-person / phone / researcher-administered]
- Duration per session: [estimated time burden on participant]
Step 5: Specify measures and instruments
For each variable being measured:
Variable: [DV or other key variable]
Instrument: [name, author, year]
Validity evidence: [what populations has this been validated with?]
Reliability: [Cronbach's α or test-retest r - cite source]
Scale: [e.g. 7-point Likert, 0-100 continuous, binary]
Scoring: [how to compute the score from items]
Interpretation: [higher scores = more/less of what?]
For qualitative:
Data type: [semi-structured interview / focus group / document / observation field notes]
Interview guide: [list of 5-8 main questions - attach as appendix]
Recording: [audio recorded with consent / field notes / etc.]
Step 6: Specify analysis plan
Write this before collecting data. Pre-registration prevents HARKing (Hypothesizing After Results are Known).
Quantitative:
Primary analysis:
Hypothesis: [restate H₁]
Statistical test: [t-test / ANOVA / regression / chi-square / etc.]
Rationale: [why this test for this data structure]
Significance threshold: α = 0.05 (or pre-specified)
Effect size: [Cohen's d / η² / r - will be reported regardless of significance]
Secondary analyses: [list]
Handling missing data: [listwise deletion / multiple imputation / etc.]
Outlier policy: [defined in advance: ±3 SD removed / winsorized / etc.]
Software: [R / SPSS / Stata / Python]
Qualitative:
Analysis method: [thematic analysis / IPA / grounded theory coding / etc.]
Coding process: [open → axial → selective coding / inductive → deductive]
Trustworthiness strategies:
- Member checking: [participants review interpretations]
- Peer debriefing: [second coder reviews sample]
- Reflexivity: [researcher positionality statement]
- Thick description: [sufficient detail for transferability judgments]
Inter-rater reliability: [if applicable - Cohen's κ target ≥ 0.70]
Step 7: Address ethical considerations
Risk level: [minimal / low / medium / high]
Key risks to participants: [list]
Mitigations: [list]
IRB status: [pending / approved - include protocol number if approved]
Informed consent: [how obtained and documented]
Data storage: [encrypted / anonymized / access controls]
Data retention: [how long kept and then how destroyed]
Conflicts of interest: [none / disclose if any]
Anti-Patterns
1. Method chosen before question Bad: "I want to do a survey" before the research question is formed. Good: Identify what you need to know, then select the method that answers it most reliably.
2. No power analysis for quantitative work Bad: "We'll recruit 50 people." Good: Power analysis using prior literature's effect size, α = 0.05, power = 0.80, gives required N, then add buffer for attrition.
3. Analysis plan written after seeing data Bad: Exploratory analysis, then framing findings as confirmatory. Good: Pre-register the primary hypothesis and analysis plan. Distinguish confirmatory from exploratory findings in the paper.
4. Instrument without validity evidence Bad: "We made a 5-question survey." Good: Use validated instruments with published reliability and validity data for the relevant population. If a novel instrument is required, include pilot validation steps.
Quality Checklist
- Research paradigm matches the research question
- Study design type named and justified
- Inclusion/exclusion criteria specified
- Sample size justified (power analysis for quantitative, saturation plan for qualitative)
- Procedure is step-by-step and replicable
- Every measured variable has a named instrument with reliability and validity evidence
- Analysis plan written before data collection (pre-registered if possible)
- Missing data, outlier, and multiple comparison policies defined in advance
- Ethical risks identified with mitigations
- IRB/ethics review status noted