Questionnaire Design Guide
Design valid and reliable survey instruments with proper question types, Likert scale construction, response coding, and data preparation for analysis.
Survey Design Principles
Question Types
| Type |
Example |
Best For |
Analysis |
| Likert scale |
"Rate your agreement: 1-5" |
Attitudes, perceptions |
Ordinal/interval statistics |
| Multiple choice |
"Select your field" |
Demographics, categories |
Frequencies, chi-square |
| Ranking |
"Rank these 5 options" |
Preferences, priorities |
Rank correlations |
| Open-ended |
"Describe your experience" |
Exploratory, rich data |
Qualitative coding |
| Matrix/grid |
Multiple items, same scale |
Efficient battery of items |
Factor analysis, reliability |
| Slider/VAS |
0-100 visual analog scale |
Continuous measures |
Parametric statistics |
| Semantic differential |
"Easy __ __ __ __ __ Difficult" |
Bipolar attitudes |
Factor analysis |
The Four C's of Good Questions
- Clear: Avoid jargon, double-barreled questions, and ambiguity
- Concise: Keep questions short (ideally under 20 words)
- Complete: Include all relevant response options
- Consistent: Use the same scale direction and format throughout
Likert Scale Design
Scale Points
| Points |
Scale Example |
Recommended Use |
| 4-point |
Strongly Disagree to Strongly Agree |
Forces choice (no neutral), less discriminating |
| 5-point |
SD, D, Neutral, A, SA |
Most common, good balance of simplicity and discrimination |
| 7-point |
SD, D, Somewhat D, Neutral, Somewhat A, A, SA |
More discriminating, better for experienced respondents |
| 11-point (0-10) |
Not at all to Completely |
NPS, continuous-like measures |
Anchoring Labels
5-Point Agreement Scale:
1 = Strongly Disagree
2 = Disagree
3 = Neither Agree nor Disagree
4 = Agree
5 = Strongly Agree
5-Point Frequency Scale:
1 = Never
2 = Rarely
3 = Sometimes
4 = Often
5 = Always
5-Point Satisfaction Scale:
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neutral
4 = Satisfied
5 = Very Satisfied
Reverse-Coded Items
Include 2-3 reverse-coded items per construct to detect acquiescence bias:
Regular: "I find research methods interesting." (1-5: SD to SA)
Reversed: "I find research methods tedious and dull." (1-5: SD to SA)
# Recode reversed items before analysis:
# reversed_score = (max_scale + 1) - raw_score
# For a 5-point scale: reversed_score = 6 - raw_score
Constructing a Multi-Item Scale
Step-by-Step Process
- Define the construct: Write a clear conceptual definition
- Generate items: Write 1.5-2x the number of items you plan to keep (e.g., write 15 items for an 8-item scale)
- Expert review: Have 3-5 experts rate each item for relevance (Content Validity Index)
- Pilot test: Administer to 30-50 respondents
- Item analysis: Calculate item-total correlations, check reliability
- Exploratory Factor Analysis (EFA): Confirm dimensionality
- Finalize scale: Remove weak items, re-test reliability
Example: Research Self-Efficacy Scale
Construct: Belief in one's ability to conduct academic research
Items (5-point Likert, Strongly Disagree to Strongly Agree):
RSE1: I can formulate clear research questions.
RSE2: I can design an appropriate research methodology.
RSE3: I can analyze data using statistical software.
RSE4: I can write a publishable research paper.
RSE5: I can critically evaluate published research.
RSE6: I can present research findings at a conference.
RSE7R: I struggle to interpret statistical results. [REVERSED]
RSE8R: I find it difficult to synthesize literature. [REVERSED]
Data Coding and Preparation
Coding Scheme
import pandas as pd
import numpy as np
# Define coding scheme
likert_coding = {
"Strongly Disagree": 1,
"Disagree": 2,
"Neither Agree nor Disagree": 3,
"Agree": 4,
"Strongly Agree": 5
}
# Apply coding
df["Q1_coded"] = df["Q1_raw"].map(likert_coding)
# Reverse code specific items
reverse_items = ["RSE7R", "RSE8R"]
max_scale = 5
for item in reverse_items:
df[f"{item}_recoded"] = (max_scale + 1) - df[item]
# Calculate composite score (mean of items)
scale_items = ["RSE1", "RSE2", "RSE3", "RSE4", "RSE5", "RSE6",
"RSE7R_recoded", "RSE8R_recoded"]
df["RSE_mean"] = df[scale_items].mean(axis=1)
Missing Data Handling
# Check missing data patterns
print(df[scale_items].isnull().sum())
print(f"Complete cases: {df[scale_items].dropna().shape[0]} / {df.shape[0]}")
# Common strategies:
# 1. Listwise deletion (if < 5% missing)
df_complete = df.dropna(subset=scale_items)
# 2. Mean imputation per item (simple but biased)
df[scale_items] = df[scale_items].fillna(df[scale_items].mean())
# 3. Person-mean imputation (if < 20% of items missing per person)
def person_mean_impute(row, items, max_missing=2):
if row[items].isnull().sum() <= max_missing:
return row[items].fillna(row[items].mean())
return row[items] # leave as NaN if too many missing
df[scale_items] = df.apply(lambda r: person_mean_impute(r, scale_items), axis=1)
Reliability Analysis
Cronbach's Alpha
import pingouin as pg
# Calculate Cronbach's alpha
alpha = pg.cronbach_alpha(df[scale_items])
print(f"Cronbach's alpha: {alpha[0]:.3f}")
# Interpretation: >= 0.70 acceptable, >= 0.80 good, >= 0.90 excellent
library(psych)
# Cronbach's alpha with item-level diagnostics
alpha_result <- alpha(data[, scale_items])
print(alpha_result)
# Check "raw_alpha if item dropped" to identify weak items
Item-Total Correlations
# Corrected item-total correlations (should be > 0.30)
item_stats <- alpha_result$item.stats
print(item_stats[, c("r.drop", "raw.alpha")])
# r.drop < 0.30: consider removing the item
# raw.alpha increases if dropped: item is weakening the scale
Validity Assessment
| Validity Type |
Method |
Criterion |
| Content validity |
Expert panel rating (CVI) |
I-CVI >= 0.78, S-CVI/Ave >= 0.90 |
| Construct validity |
Exploratory Factor Analysis (EFA) |
Eigenvalue > 1, loadings > 0.40 |
| Convergent validity |
Correlation with related construct |
r > 0.30 |
| Discriminant validity |
Correlation with unrelated construct |
r < 0.30 |
| Criterion validity |
Correlation with external criterion |
Significant correlation |
| Test-retest reliability |
ICC or Pearson r over 2-4 weeks |
ICC > 0.70 |
Common Design Mistakes
| Mistake |
Example |
Fix |
| Double-barreled question |
"This course is interesting and useful" |
Split into two separate items |
| Leading question |
"Don't you agree that X is important?" |
"How important is X to you?" |
| Absolute terms |
"Do you always check citations?" |
"How often do you check citations?" |
| Missing option |
No "Not Applicable" when needed |
Add N/A option or filter logic |
| Inconsistent scale direction |
Some items 1=good, others 1=bad |
Standardize direction; clearly mark reversed items |
| Too many items |
100-item survey |
Aim for 5-8 items per construct, 15-30 min total |
| No pilot test |
Skip straight to full deployment |
Always pilot with 30-50 respondents |
Survey Platform Comparison
| Platform |
Cost |
Features |
Best For |
| Qualtrics |
Institutional |
Advanced logic, panels, API |
Large academic studies |
| SurveyMonkey |
Freemium |
Easy to use, basic analysis |
Quick surveys |
| Google Forms |
Free |
Simple, integrates with Sheets |
Classroom, pilot testing |
| LimeSurvey |
Free/self-hosted |
Open source, full control |
Privacy-sensitive research |
| REDCap |
Free (academic) |
Clinical data, HIPAA compliant |
Medical/clinical research |
| Prolific |
Per-response |
Participant recruitment |
Online experiments |
1---
2name: questionnaire-design-guide
3description: "Questionnaire and survey design with Likert scales and coding"
4metadata:
5 openclaw:
6 emoji: "📋"
7 category: "analysis"
8 subcategory: "wrangling"
9 keywords: ["questionnaire design", "survey design", "Likert scale", "data transformation"]
10 source: https://skillmd.com/skills/brycewang-stanford/questionnaire-design-guide
11 upstream: "wentor-research-plugins"
12---
13
14# Questionnaire Design Guide
15
16Design valid and reliable survey instruments with proper question types, Likert scale construction, response coding, and data preparation for analysis.
17
18## Survey Design Principles
19
20### Question Types
21
22| Type | Example | Best For | Analysis |
23|------|---------|----------|----------|
24| **Likert scale** | "Rate your agreement: 1-5" | Attitudes, perceptions | Ordinal/interval statistics |
25| **Multiple choice** | "Select your field" | Demographics, categories | Frequencies, chi-square |
26| **Ranking** | "Rank these 5 options" | Preferences, priorities | Rank correlations |
27| **Open-ended** | "Describe your experience" | Exploratory, rich data | Qualitative coding |
28| **Matrix/grid** | Multiple items, same scale | Efficient battery of items | Factor analysis, reliability |
29| **Slider/VAS** | 0-100 visual analog scale | Continuous measures | Parametric statistics |
30| **Semantic differential** | "Easy __ __ __ __ __ Difficult" | Bipolar attitudes | Factor analysis |
31
32### The Four C's of Good Questions
33
341. **Clear**: Avoid jargon, double-barreled questions, and ambiguity
352. **Concise**: Keep questions short (ideally under 20 words)
363. **Complete**: Include all relevant response options
374. **Consistent**: Use the same scale direction and format throughout
38
39## Likert Scale Design
40
41### Scale Points
42
43| Points | Scale Example | Recommended Use |
44|--------|---------------|-----------------|
45| 4-point | Strongly Disagree to Strongly Agree | Forces choice (no neutral), less discriminating |
46| 5-point | SD, D, Neutral, A, SA | Most common, good balance of simplicity and discrimination |
47| 7-point | SD, D, Somewhat D, Neutral, Somewhat A, A, SA | More discriminating, better for experienced respondents |
48| 11-point (0-10) | Not at all to Completely | NPS, continuous-like measures |
49
50### Anchoring Labels
51
52```
535-Point Agreement Scale:
541 = Strongly Disagree
552 = Disagree
563 = Neither Agree nor Disagree
574 = Agree
585 = Strongly Agree
59
605-Point Frequency Scale:
611 = Never
622 = Rarely
633 = Sometimes
644 = Often
655 = Always
66
675-Point Satisfaction Scale:
681 = Very Dissatisfied
692 = Dissatisfied
703 = Neutral
714 = Satisfied
725 = Very Satisfied
73```
74
75### Reverse-Coded Items
76
77Include 2-3 reverse-coded items per construct to detect acquiescence bias:
78
79```
80Regular: "I find research methods interesting." (1-5: SD to SA)
81Reversed: "I find research methods tedious and dull." (1-5: SD to SA)
82
83# Recode reversed items before analysis:
84# reversed_score = (max_scale + 1) - raw_score
85# For a 5-point scale: reversed_score = 6 - raw_score
86```
87
88## Constructing a Multi-Item Scale
89
90### Step-by-Step Process
91
921. **Define the construct**: Write a clear conceptual definition
932. **Generate items**: Write 1.5-2x the number of items you plan to keep (e.g., write 15 items for an 8-item scale)
943. **Expert review**: Have 3-5 experts rate each item for relevance (Content Validity Index)
954. **Pilot test**: Administer to 30-50 respondents
965. **Item analysis**: Calculate item-total correlations, check reliability
976. **Exploratory Factor Analysis (EFA)**: Confirm dimensionality
987. **Finalize scale**: Remove weak items, re-test reliability
99
100### Example: Research Self-Efficacy Scale
101
102```
103Construct: Belief in one's ability to conduct academic research
104
105Items (5-point Likert, Strongly Disagree to Strongly Agree):
106RSE1: I can formulate clear research questions.
107RSE2: I can design an appropriate research methodology.
108RSE3: I can analyze data using statistical software.
109RSE4: I can write a publishable research paper.
110RSE5: I can critically evaluate published research.
111RSE6: I can present research findings at a conference.
112RSE7R: I struggle to interpret statistical results. [REVERSED]
113RSE8R: I find it difficult to synthesize literature. [REVERSED]
114```
115
116## Data Coding and Preparation
117
118### Coding Scheme
119
120```python
121import pandas as pd
122import numpy as np
123
124# Define coding scheme
125likert_coding = {
126 "Strongly Disagree": 1,
127 "Disagree": 2,
128 "Neither Agree nor Disagree": 3,
129 "Agree": 4,
130 "Strongly Agree": 5
131}
132
133# Apply coding
134df["Q1_coded"] = df["Q1_raw"].map(likert_coding)
135
136# Reverse code specific items
137reverse_items = ["RSE7R", "RSE8R"]
138max_scale = 5
139for item in reverse_items:
140 df[f"{item}_recoded"] = (max_scale + 1) - df[item]
141
142# Calculate composite score (mean of items)
143scale_items = ["RSE1", "RSE2", "RSE3", "RSE4", "RSE5", "RSE6",
144 "RSE7R_recoded", "RSE8R_recoded"]
145df["RSE_mean"] = df[scale_items].mean(axis=1)
146```
147
148### Missing Data Handling
149
150```python
151# Check missing data patterns
152print(df[scale_items].isnull().sum())
153print(f"Complete cases: {df[scale_items].dropna().shape[0]} / {df.shape[0]}")
154
155# Common strategies:
156# 1. Listwise deletion (if < 5% missing)
157df_complete = df.dropna(subset=scale_items)
158
159# 2. Mean imputation per item (simple but biased)
160df[scale_items] = df[scale_items].fillna(df[scale_items].mean())
161
162# 3. Person-mean imputation (if < 20% of items missing per person)
163def person_mean_impute(row, items, max_missing=2):
164 if row[items].isnull().sum() <= max_missing:
165 return row[items].fillna(row[items].mean())
166 return row[items] # leave as NaN if too many missing
167
168df[scale_items] = df.apply(lambda r: person_mean_impute(r, scale_items), axis=1)
169```
170
171## Reliability Analysis
172
173### Cronbach's Alpha
174
175```python
176import pingouin as pg
177
178# Calculate Cronbach's alpha
179alpha = pg.cronbach_alpha(df[scale_items])
180print(f"Cronbach's alpha: {alpha[0]:.3f}")
181# Interpretation: >= 0.70 acceptable, >= 0.80 good, >= 0.90 excellent
182```
183
184```r
185library(psych)
186
187# Cronbach's alpha with item-level diagnostics
188alpha_result <- alpha(data[, scale_items])
189print(alpha_result)
190# Check "raw_alpha if item dropped" to identify weak items
191```
192
193### Item-Total Correlations
194
195```r
196# Corrected item-total correlations (should be > 0.30)
197item_stats <- alpha_result$item.stats
198print(item_stats[, c("r.drop", "raw.alpha")])
199# r.drop < 0.30: consider removing the item
200# raw.alpha increases if dropped: item is weakening the scale
201```
202
203## Validity Assessment
204
205| Validity Type | Method | Criterion |
206|--------------|--------|-----------|
207| **Content validity** | Expert panel rating (CVI) | I-CVI >= 0.78, S-CVI/Ave >= 0.90 |
208| **Construct validity** | Exploratory Factor Analysis (EFA) | Eigenvalue > 1, loadings > 0.40 |
209| **Convergent validity** | Correlation with related construct | r > 0.30 |
210| **Discriminant validity** | Correlation with unrelated construct | r < 0.30 |
211| **Criterion validity** | Correlation with external criterion | Significant correlation |
212| **Test-retest reliability** | ICC or Pearson r over 2-4 weeks | ICC > 0.70 |
213
214## Common Design Mistakes
215
216| Mistake | Example | Fix |
217|---------|---------|-----|
218| Double-barreled question | "This course is interesting and useful" | Split into two separate items |
219| Leading question | "Don't you agree that X is important?" | "How important is X to you?" |
220| Absolute terms | "Do you always check citations?" | "How often do you check citations?" |
221| Missing option | No "Not Applicable" when needed | Add N/A option or filter logic |
222| Inconsistent scale direction | Some items 1=good, others 1=bad | Standardize direction; clearly mark reversed items |
223| Too many items | 100-item survey | Aim for 5-8 items per construct, 15-30 min total |
224| No pilot test | Skip straight to full deployment | Always pilot with 30-50 respondents |
225
226## Survey Platform Comparison
227
228| Platform | Cost | Features | Best For |
229|----------|------|----------|----------|
230| Qualtrics | Institutional | Advanced logic, panels, API | Large academic studies |
231| SurveyMonkey | Freemium | Easy to use, basic analysis | Quick surveys |
232| Google Forms | Free | Simple, integrates with Sheets | Classroom, pilot testing |
233| LimeSurvey | Free/self-hosted | Open source, full control | Privacy-sensitive research |
234| REDCap | Free (academic) | Clinical data, HIPAA compliant | Medical/clinical research |
235| Prolific | Per-response | Participant recruitment | Online experiments |