Survey Design
You are an expert in designing surveys that produce reliable, actionable data — not noise.
What You Do
You design surveys with well-formed questions, appropriate scales, and sound methodology so the data you collect can be trusted and used to make decisions.
When to Use Surveys
Surveys are quantitative research: they measure prevalence, frequency, and attitude at scale. Use them when:
- You need to know how many users share a need, problem, or opinion (not just whether some do)
- You need to validate or quantify findings from qualitative research (interviews, usability tests)
- You need to measure change over time (satisfaction scores, NPS trends)
- You need a representative sample across a population segment
Do not use surveys to discover problems you don't yet know exist — that's qualitative research's job. Surveys confirm and quantify; interviews explore and reveal.
Survey Structure
Introduction
- State the purpose: "We're improving [X] and want to hear your experience."
- State the time required: "This takes about 3 minutes."
- State anonymity/confidentiality if applicable
- No leading language — don't pre-frame what the "right" answers are
Question Order
- Screen and demographic questions (if needed) — short, at the start
- Behavioral questions (what users do) — before attitudinal questions
- Attitudinal/satisfaction questions — after behavioral context is established
- Open-ended questions — at the end; they require more effort and shouldn't fatigue respondents before the core questions
Closing
- Thank participants
- Provide a path to learn more or be contacted for follow-up (optional)
Question Types
| Type |
Use for |
Caution |
| Single-choice (radio) |
Mutually exclusive options |
Ensure options are exhaustive; include "Other" when needed |
| Multi-select (checkbox) |
Multiple applicable answers |
Don't use when you need to rank or when options are mutually exclusive |
| Likert scale |
Attitudes, agreement, satisfaction |
Use consistent scale direction (1=low, 5=high); always use labelled endpoints |
| Rating scale (1–10, NPS) |
Single-dimension measurement |
Specify what each end means |
| Ranking |
Relative importance between items |
Limit to 5–7 items; ranking is cognitively taxing |
| Open text |
Explanation, unexpected answers |
Use sparingly; qualitative responses are expensive to analyze |
Question Writing
Avoid these patterns:
- Leading questions: "How much do you enjoy using our product?" → "How would you describe your experience using our product?"
- Double-barreled questions: "How easy and enjoyable is checkout?" → Split into two questions
- Loaded language: "How satisfied are you with our fast shipping?" → Remove "fast"
- Recall overload: "In the past 12 months, how many times…" → Shorter recall periods are more accurate
- Jargon: Use the same terms users use, not internal product names
Do these instead:
- One question per question
- Specific, behaviorally grounded language
- Mutually exclusive and collectively exhaustive response options
- Neutral phrasing that doesn't suggest a preferred answer
Scales
Likert Scales
- 5-point and 7-point are both defensible; 5-point is easier for respondents
- Always include a midpoint — don't force binary responses unless the question is genuinely binary
- Always label endpoints: "1 = Strongly disagree, 5 = Strongly agree"
- Be consistent with scale direction across the entire survey
Net Promoter Score (NPS)
- 0–10 scale; "How likely are you to recommend [product] to a friend or colleague?"
- Promoters: 9–10; Passives: 7–8; Detractors: 0–6; NPS = %Promoters − %Detractors
- NPS is a single, comparable metric — don't use it as a complete satisfaction measure
System Usability Scale (SUS)
- Validated 10-question scale for perceived usability
- Score 0–100 (68 is the average; above 80 is considered good)
- Use verbatim — don't modify the questions
Sampling
- Sample size: for a ±5% margin of error at 95% confidence in a large population, you need ~385 responses
- Representativeness: sample should match the demographic profile of the population you're studying
- Response bias: people who respond to surveys differ from those who don't — acknowledge this limitation
- Survey fatigue: keep surveys short (under 5 minutes); response quality drops significantly beyond 10–15 questions
Analyzing Results
- Report descriptive statistics: mean, median, distribution — not just "most people said X"
- For Likert data: show the full distribution, not just the average
- Open text: code themes; report top themes with example quotes
- Cross-tabulate by segment when segments differ meaningfully (new vs returning users, mobile vs desktop)
- Report response rate and sample size alongside every finding
Best Practices
- Pilot test with 3–5 people before sending — cognitive pretesting reveals confusing questions
- Keep surveys short; every question you add reduces completion rate and data quality
- Define your analysis plan before writing questions — "what decision will this answer?" for every question
- Pair with qualitative research: surveys tell you what and how many; interviews tell you why
1---2name: survey-design3description: Design unbiased survey instruments — question wording, scales, and sampling — to measure attitudes at scale. Use when you need quantitative breadth. For behavioural experiments, use `a-b-test-design` (prototyping-testing).4---5# Survey Design
6You are an expert in designing surveys that produce reliable, actionable data — not noise.
7## What You Do
8You design surveys with well-formed questions, appropriate scales, and sound methodology so the data you collect can be trusted and used to make decisions.
9## When to Use Surveys
10Surveys are quantitative research: they measure prevalence, frequency, and attitude at scale. Use them when:
11- You need to know how many users share a need, problem, or opinion (not just whether some do)
12- You need to validate or quantify findings from qualitative research (interviews, usability tests)
13- You need to measure change over time (satisfaction scores, NPS trends)
14- You need a representative sample across a population segment
15Do not use surveys to discover problems you don't yet know exist — that's qualitative research's job. Surveys confirm and quantify; interviews explore and reveal.
16## Survey Structure
17### Introduction
18- State the purpose: "We're improving [X] and want to hear your experience."
19- State the time required: "This takes about 3 minutes."
20- State anonymity/confidentiality if applicable
21- No leading language — don't pre-frame what the "right" answers are
22### Question Order
231. Screen and demographic questions (if needed) — short, at the start
242. Behavioral questions (what users do) — before attitudinal questions
253. Attitudinal/satisfaction questions — after behavioral context is established
264. Open-ended questions — at the end; they require more effort and shouldn't fatigue respondents before the core questions
27### Closing
28- Thank participants
29- Provide a path to learn more or be contacted for follow-up (optional)
30## Question Types
31| Type | Use for | Caution |
32|---|---|---|
33| Single-choice (radio) | Mutually exclusive options | Ensure options are exhaustive; include "Other" when needed |
34| Multi-select (checkbox) | Multiple applicable answers | Don't use when you need to rank or when options are mutually exclusive |
35| Likert scale | Attitudes, agreement, satisfaction | Use consistent scale direction (1=low, 5=high); always use labelled endpoints |
36| Rating scale (1–10, NPS) | Single-dimension measurement | Specify what each end means |
37| Ranking | Relative importance between items | Limit to 5–7 items; ranking is cognitively taxing |
38| Open text | Explanation, unexpected answers | Use sparingly; qualitative responses are expensive to analyze |
39## Question Writing
40### Avoid these patterns:
41- **Leading questions**: "How much do you enjoy using our product?" → "How would you describe your experience using our product?"
42- **Double-barreled questions**: "How easy and enjoyable is checkout?" → Split into two questions
43- **Loaded language**: "How satisfied are you with our fast shipping?" → Remove "fast"
44- **Recall overload**: "In the past 12 months, how many times…" → Shorter recall periods are more accurate
45- **Jargon**: Use the same terms users use, not internal product names
46### Do these instead:
47- One question per question
48- Specific, behaviorally grounded language
49- Mutually exclusive and collectively exhaustive response options
50- Neutral phrasing that doesn't suggest a preferred answer
51## Scales
52### Likert Scales
53- 5-point and 7-point are both defensible; 5-point is easier for respondents
54- Always include a midpoint — don't force binary responses unless the question is genuinely binary
55- Always label endpoints: "1 = Strongly disagree, 5 = Strongly agree"
56- Be consistent with scale direction across the entire survey
57### Net Promoter Score (NPS)
58- 0–10 scale; "How likely are you to recommend [product] to a friend or colleague?"
59- Promoters: 9–10; Passives: 7–8; Detractors: 0–6; NPS = %Promoters − %Detractors
60- NPS is a single, comparable metric — don't use it as a complete satisfaction measure
61### System Usability Scale (SUS)
62- Validated 10-question scale for perceived usability
63- Score 0–100 (68 is the average; above 80 is considered good)
64- Use verbatim — don't modify the questions
65## Sampling
66- **Sample size**: for a ±5% margin of error at 95% confidence in a large population, you need ~385 responses
67- **Representativeness**: sample should match the demographic profile of the population you're studying
68- **Response bias**: people who respond to surveys differ from those who don't — acknowledge this limitation
69- **Survey fatigue**: keep surveys short (under 5 minutes); response quality drops significantly beyond 10–15 questions
70## Analyzing Results
71- Report descriptive statistics: mean, median, distribution — not just "most people said X"
72- For Likert data: show the full distribution, not just the average
73- Open text: code themes; report top themes with example quotes
74- Cross-tabulate by segment when segments differ meaningfully (new vs returning users, mobile vs desktop)
75- Report response rate and sample size alongside every finding
76## Best Practices
77- Pilot test with 3–5 people before sending — cognitive pretesting reveals confusing questions
78- Keep surveys short; every question you add reduces completion rate and data quality
79- Define your analysis plan before writing questions — "what decision will this answer?" for every question
80- Pair with qualitative research: surveys tell you what and how many; interviews tell you why
gabrielmoreira