Study Design Checklist

Study Design Checklist

Overview

Good analysis starts with good study design. This checklist ensures key design elements are addressed before analysis begins.

Research Question Formulation

PICO(T) Framework

Question Checklist

• Is the question clearly stated in plain language?
• Is it answerable with available data?
• Is it clinically meaningful?
• Is it novel or confirmatory of prior work?

Cohort Definition

Inclusion Criteria

Define who enters the study:

Exclusion Criteria

Define who is removed (with justification):

Documentation Requirements

For each criterion, document:

• Exact definition (codes, thresholds, timing)
• Number excluded at each step
• Justify each exclusion

CONSORT-style flow diagram recommended.

Unit of Analysis

Options

Key Decision: Multiple Events per Patient

If patients can have multiple admissions/stays:

Time-Zero and Follow-Up

Time-Zero Definition

Time-zero is the moment when:

• Follow-up begins
• Patient becomes "at risk" for outcome
• Exposure status should be determined (or treatment assigned)

Follow-Up Period

• Start: Time-zero
• End: Outcome, death, discharge, loss to follow-up, or administrative censoring
• Fixed horizon: e.g., 30 days, 90 days, hospital stay

Censoring

Define what ends follow-up without outcome:

• Discharge alive
• Transfer to another facility
• End of study period
• Loss of data availability

Consider: Is censoring informative? (Related to outcome?)

Exposure / Predictor Definition

For Observational Studies

Avoiding Data Leakage

For prediction models:

• Features must be available at prediction time
• Cannot use information that comes after time-zero
• Cannot use information that requires knowing the outcome

Red flags:

• Using discharge diagnoses to predict at admission
• Using lab values from after the event
• Using length of stay to predict mortality

Outcome Definition

Characteristics of Good Outcomes

Common ICU Outcomes

Competing Events

Some outcomes have competing events:

• Studying ICU discharge → death is competing event
• Studying readmission → death prevents readmission

Options:

• Composite outcome (either event)
• Competing risk analysis (Fine-Gray)
• Cause-specific analysis

Confounding

Identifying Confounders

A confounder is a variable that:

1. Is associated with the exposure
2. Is associated with the outcome
3. Is NOT on the causal pathway between exposure and outcome

Common Confounder Categories in ICU Research

Adjustment Strategies

Unmeasured Confounding

Problem: Can never prove no unmeasured confounding in observational data.

Mitigation:

• Be explicit about assumed confounders
• Use negative control outcomes
• Quantitative bias analysis (E-value)
• Discuss plausible unmeasured confounders

Sensitivity Analysis Planning

Pre-Specified Sensitivity Analyses

Plan before looking at data:

Bias Quantification

For causal inference:

• E-value: How strong would unmeasured confounding need to be to nullify the result?
• Probabilistic bias analysis: Model uncertainty about bias

Power and Sample Size

Pre-Study Considerations

Before starting, determine:

• Minimum clinically important effect size
• Expected event rate
• Required sample size for adequate power

Rules of Thumb

What If Underpowered?

• Acknowledge limitation
• Report effect size and CI (even if non-significant)
• Consider as hypothesis-generating

Reproducibility Checklist

Before finalizing design, ensure:

• Research question is clearly stated (PICO format)
• Population is defined with explicit inclusion/exclusion
• Unit of analysis is specified
• Time-zero is clearly defined
• Follow-up period is specified
• Outcome is objectively defined
• Exposure/predictors are defined and temporally appropriate
• Confounders are identified with adjustment plan
• Missing data handling is planned
• Sensitivity analyses are pre-specified
• Sample size is adequate for planned analyses

Common Design Pitfalls