Survey Design

Survey Design Patterns

Survey Architecture

Survey Structure

1. Introduction (required)
   - Who you are, why you're asking, how long it takes
   - Anonymity/confidentiality statement
   - Incentive (if any)

2. Screener questions (optional)
   - Qualify respondents before going deep
   - Use skip logic to exit non-qualifiers early

3. Warm-up questions (2-3)
   - Easy, non-threatening
   - Related to topic but not the core issue

4. Core questions (main body)
   - Most important questions first
   - Most sensitive/demographic last

5. Open-ended (1-3)
   - "Anything else you'd like to share?"
   - Keep optional

6. Demographic questions (end)
   - Age, role, company size, etc.
   - Only ask what you'll actually use

Question Types

Closed Question Types

Likert Scale Design

5-point: Strongly Disagree / Disagree / Neutral / Agree / Strongly Agree
7-point: Adds "Somewhat" variants for more nuance
Use 7-point for research requiring finer distinctions

Label ALL points (not just ends) — anchors reduce variability

Balanced scale (equal + and - options):
Strongly Disagree — Disagree — Neither — Agree — Strongly Agree

Avoid:
- Unbalanced: Poor / Fair / Good / Very Good / Excellent (positive skew)
- Unless measuring satisfaction (acceptable to use positive-anchored scale)

NPS Question

"How likely are you to recommend [product] to a friend or colleague?"
Scale: 0 (Not at all likely) — 10 (Extremely likely)

Segmentation:
0-6: Detractors
7-8: Passives
9-10: Promoters

NPS = % Promoters − % Detractors
Range: −100 to +100

Benchmarks (SaaS):
< 0: Danger zone
0-30: Room for improvement
30-70: Good
> 70: Excellent (Apple ~72, Slack ~51)

Bias Prevention

Common Survey Biases

Question Writing Rules

✅ Good:
"How often do you use [feature]?"
  ○ Daily   ○ Weekly   ○ Monthly   ○ Rarely   ○ Never

❌ Bad:
"Don't you think [feature] makes things easier?"  ← leading
"How often do you use this great feature?"  ← loaded
"How often do you use [feature] and would you recommend it?"  ← double-barreled

✅ Good (sensitive topic):
"Some people feel [behavior A] while others feel [behavior B]. 
 Which is closer to your experience?"

❌ Bad:
"Do you ever [embarrassing behavior]?"

Sampling Strategy

Probability Sampling (for generalization)

Non-Probability Sampling (for exploration)

Sample Size Calculator

import math

def sample_size(population, confidence=0.95, margin_error=0.05, p=0.5):
    """
    population: total population size (use 1e9 for unknown)
    confidence: 0.90, 0.95, or 0.99
    margin_error: desired margin of error (0.05 = ±5%)
    p: expected proportion (0.5 = most conservative)
    """
    z_scores = {0.90: 1.645, 0.95: 1.96, 0.99: 2.576}
    z = z_scores[confidence]
    
    n_inf = (z**2 * p * (1-p)) / margin_error**2
    n = n_inf / (1 + (n_inf - 1) / population)
    
    return math.ceil(n)

# Examples
print(sample_size(10000))    # 370 for 95% CI ±5%
print(sample_size(100000))   # 383 for 95% CI ±5%
print(sample_size(1e9))      # 385 (large population asymptote)

Response Rate Optimization

Response Rate Benchmarks

Improving Response Rate

• Keep it short — 5 min target, show progress bar
• Time it right — after successful interaction, not mid-task
• Mobile-optimize — >60% open email on mobile
• Personalize subject — "[Name], 2 min to improve [product]"
• Pre-notify — "Next week we'll ask you about [topic]"
• Incentivize — gift card, donation, exclusive content
• Follow up once — single reminder 3-5 days later
• Executive signature — CEO name increases open rate for B2B

Analysis and Reporting

Likert Analysis Approaches

Approach 1: Treat as ordinal (conservative)
- Use median and IQR
- Non-parametric tests (Mann-Whitney, Kruskal-Wallis)

Approach 2: Treat as interval (common in practice)
- Use mean and std dev
- Visualize with stacked bar (% favorable vs unfavorable)

Top-2-box score:
% who chose top 2 responses (Agree + Strongly Agree)
Most actionable for tracking over time

Survey Report Template

## Key Findings

### Executive Summary
[3 bullet points: what you found, what it means, what to do]

### NPS / Satisfaction Score
Current: X | Previous: Y | Change: +Z pts
Benchmark: Industry avg XX

### Top Pain Points
1. [Theme] — mentioned by X% of respondents
2. [Theme] — mentioned by X% of respondents

### Verbatim Highlights
"[Best positive quote]" — [Persona type]
"[Most actionable critical quote]" — [Persona type]

### Recommendations
1. [Action] — addresses [% of respondents who cited this]
2. [Action]

### Methodology
- N = X respondents
- Dates: [range]
- Channel: [email/in-app/etc]
- Response rate: X%