Survey Design — Survey & Instrument Design Agent
A comprehensive survey and instrument design tool for faculty and researchers. Covers the full lifecycle of survey-based research: from construct definition and item writing through pilot testing, validation, deployment, and analysis of survey data.
Overview
Survey research is one of the most widely used methods across social sciences, health sciences, education, and business. Despite its apparent simplicity, designing a valid and reliable survey instrument requires systematic attention to construct definition, item wording, response format, sampling, bias mitigation, and psychometric validation.
This skill treats survey design as a scientific process, not an art. Every design decision should be justified and documented.
When to Use This Skill
This skill should be used when:
- Designing a new survey or questionnaire from scratch
- Adapting an existing instrument for a new population or context
- Writing Likert-scale items or other structured response formats
- Developing interview protocols or focus group guides
- Planning sampling strategies for survey research
- Conducting pilot tests and cognitive interviews
- Validating instruments (reliability and validity analysis)
- Selecting online survey platforms (Qualtrics, REDCap, Google Forms)
- Improving response rates and reducing bias
- Conducting cultural adaptation and translation of instruments
- Teaching research methods courses that include survey design
Does NOT Trigger
| Scenario |
Use Instead |
| Qualitative data analysis (coding, themes, focus group/interview analysis) |
alterlab-qualitative-methods |
| Integrating qual + quant strands (convergent/sequential designs, joint displays) |
alterlab-mixed-methods |
| Hypothesis-test selection, assumption checks, power analysis beyond validation |
alterlab-statistical-analysis |
| Specialized social-science methods (Delphi, Q-methodology, QCA) |
alterlab-social-science-methods |
| Writing the research paper |
alterlab-paper-writer |
| Ethics/IRB applications, informed consent for survey research |
alterlab-research-ethics |
Core Capabilities
1. Survey Design Process
The 10-Step Survey Design Framework:
Step 1: Define research objectives and constructs
What do you want to measure? What are your research questions?
│
Step 2: Review existing instruments
Has someone already validated an instrument for your construct?
│
Step 3: Define the target population and sampling frame
Who will you survey? How will you reach them?
│
Step 4: Choose survey mode
Online, paper, phone, in-person, mixed-mode?
│
Step 5: Write items and response options
Craft questions that are clear, unambiguous, and aligned to constructs
│
Step 6: Design survey structure and flow
Organize sections, add skip logic, manage survey length
│
Step 7: Expert review
Subject matter experts and methodologists evaluate the instrument
│
Step 8: Cognitive interviews and pilot testing
Test with a small sample from the target population
│
Step 9: Psychometric validation
Reliability analysis, factor analysis, validity assessment
│
Step 10: Deploy, monitor, and analyze
Launch survey, track response rates, clean and analyze data
2. Construct Definition and Operationalization
Before writing a single item, define what you are measuring. Map each construct to its dimensions, indicators, distinct-but-related constructs, and a nomological network so every item traces back to something specific.
Full Construct Mapping Template: see references/item_writing_and_scales.md.
3. Item Writing
Question Types and When to Use Them
| Type |
Format |
Best For |
| Closed-ended (single choice) |
Radio buttons |
Mutually exclusive categories |
| Closed-ended (multiple choice) |
Checkboxes |
Non-mutually exclusive categories |
| Likert scale |
Rating scale |
Attitudes, perceptions, frequency |
| Semantic differential |
Bipolar scale |
Evaluative judgments |
| Ranking |
Drag-and-drop or numbered |
Forced prioritization |
| Matrix/Grid |
Likert items in table |
Multiple items with same response scale |
| Open-ended |
Text box |
Exploratory, rich responses |
| Numeric |
Number input |
Precise quantities |
| Visual analog scale (VAS) |
Slider |
Continuous measurement |
Likert Scale — Number of Points
| Points |
Trade-off |
Use When |
| 4-point |
Forces a choice (no midpoint) |
Avoid social desirability midpoint clustering |
| 5-point |
Most common, well-understood; central-tendency bias |
Standard attitudinal measurement |
| 6-point |
Forced choice with more granularity |
Force direction with more options |
| 7-point |
Greater discrimination; better for factor analysis |
Established psychometric instruments |
Item-writing essentials — DO: simple/clear language; one concept per item (no double-barreled); specific time frames; match scale to stem; pilot with the target population; over-generate items. DO NOT: leading/loaded language; double negatives; assume knowledge; use absolutes; write overly long items; ask about hypotheticals when you mean actual behavior.
Full question-type examples, Likert labeling schemes, the complete DO / DO NOT rules, and worked before/after item revisions: see references/item_writing_and_scales.md.
4. Survey Structure and Flow
Organize the instrument as: welcome + consent → screening → main content grouped by construct (easy questions first, sensitive items mid-survey) → demographics at the end → thank-you/debrief. Use skip logic to hide irrelevant questions and route ineligible respondents.
Full Survey Structure Template and Skip Logic Design examples: see references/item_writing_and_scales.md.
5. Sampling Strategies
Probability Sampling (every member has a known, non-zero chance of selection; generalizable):
| Method |
How It Works |
Trade-off |
| Simple random |
Select randomly from complete list |
Unbiased, but needs a complete sampling frame |
| Systematic |
Select every kth element |
Easy, but periodicity risk if list has a pattern |
| Stratified |
Random sample within population strata |
Ensures subgroup representation; needs population knowledge |
| Cluster |
Randomly select clusters, then sample within |
Practical without individual list; higher sampling error |
| Multi-stage |
Combine methods (cluster then stratified) |
Flexible for large populations; complex to analyze |
Non-Probability Sampling (no representativeness guarantee):
| Method |
How It Works |
Trade-off |
| Convenience |
Recruit whoever is available |
Fast/cheap, but strong bias |
| Purposive |
Select on specific criteria |
Targets relevant subgroups; researcher bias |
| Snowball |
Participants recruit others |
Reaches hidden populations; biased toward the connected |
| Quota |
Convenience sample within subgroup quotas |
Ensures diversity; not truly random within quotas |
Size the sample with the proportion formula n = (Z² × p × (1-p)) / E² for descriptive surveys (adjusting for finite population and expected response rate), or a power analysis for comparative surveys. Worked sample-size formulas and the Python two-group power-analysis helper: see references/sampling_and_power.md.
6. Response Bias Mitigation
| Bias Type |
Definition |
Mitigation Strategies |
| Social desirability |
Respondents answer in ways they believe are socially acceptable |
Anonymous data collection; indirect questioning; validated social desirability scales (e.g., Marlowe-Crowne) |
| Acquiescence |
Tendency to agree with statements regardless of content |
Mix positively and negatively worded items; use forced-choice formats |
| Central tendency |
Tendency to select middle response options |
Use even-point scales (no midpoint); provide behavioral anchors |
| Extreme responding |
Tendency to select extreme endpoints |
Use more response options (7-point); provide clear anchor descriptions |
| Order effects |
Earlier questions influence responses to later questions |
Randomize item order within sections; counterbalance across respondents |
| Nonresponse bias |
Systematic differences between responders and non-responders |
Follow-up reminders; analyze early vs. late responders; compare demographics to population |
| Recall bias |
Inaccurate recall of past events |
Use shorter recall periods; provide memory aids; use event-specific prompts |
| Common method bias |
Inflated correlations due to same measurement method |
Use different measurement methods; temporal separation; marker variables |
7. Pilot Testing
Run a three-phase pilot before full deployment: (1) expert review for content/face validity (CVI thresholds: Item-CVI ≥ 0.78, Scale-CVI/Ave ≥ 0.90); (2) cognitive interviews (n = 5-10) using think-aloud and probing questions; (3) a quantitative pilot (n = 30-50) assessing completion, missing data, distributions, internal consistency, and item-total correlations.
Full phase-by-phase protocol with probe scripts and the quantitative-pilot checklist: see references/pilot_and_validation.md.
8. Instrument Validation
Assess reliability (Cronbach's alpha per subscale, corrected item-total correlations — flag items < 0.30) and validity across the evidence types below. Use exploratory factor analysis (Bartlett's test, KMO, eigenvalues, rotated loadings) to check internal structure.
| Type |
Question |
Method |
| Content validity |
Do items cover the construct adequately? |
Expert review, CVI calculation |
| Face validity |
Do items appear to measure the construct? |
Target population review |
| Construct validity |
Does it measure the theoretical construct? |
Factor analysis (EFA/CFA) |
| Convergent validity |
Does it correlate with similar measures? |
Correlation with established instruments (r > 0.50) |
| Discriminant validity |
Is it distinct from different constructs? |
Low correlation with unrelated measures (r < 0.30) |
| Criterion (concurrent) |
Does it correlate with a current criterion? |
Correlation with gold standard, measured simultaneously |
| Criterion (predictive) |
Does it predict a future outcome? |
Correlation with criterion measured later |
| Known-groups |
Can it distinguish groups known to differ? |
Compare scores between groups that should differ |
Runnable Python for Cronbach's alpha, item-total correlations, EFA, and the three-phase pilot protocol: see references/pilot_and_validation.md. For CFA fit indices, measurement invariance, and the broader psychometric framework, see references/survey-methodology.md.
9. Online Survey Platform Comparison
| Feature |
Qualtrics |
REDCap |
Google Forms |
LimeSurvey |
| Cost |
Institutional license (expensive) |
Free for institutions |
Free |
Free (open source) |
| Skip logic |
Advanced |
Advanced |
Basic |
Advanced |
| Randomization |
Yes (items, blocks) |
Limited |
No |
Yes |
| Piping |
Yes |
Yes |
No |
Yes |
| Offline data collection |
Yes (app) |
Yes (app) |
No |
Yes |
| HIPAA compliant |
Yes (BAA available) |
Yes (designed for it) |
No |
Self-hosted: yes |
| API access |
Yes |
Yes |
Limited |
Yes |
| Data export |
CSV, SPSS, Excel |
CSV, Excel, SPSS, SAS, R, Stata |
CSV, Excel |
CSV, Excel, SPSS, R |
| Multi-language |
Yes |
Yes |
Manual |
Yes |
| Panel integration |
Yes (Prolific, MTurk) |
No |
No |
Limited |
| Best for |
Complex academic surveys |
Clinical and health research |
Simple surveys, course evaluations |
Budget-conscious complex surveys |
10. Interview Protocol Development
Build semi-structured interview guides with a scripted opening/consent, a warm-up question, main-question blocks organized by construct (each with probes), a closing catch-all, and a post-interview field-notes routine.
Full Semi-Structured Interview Guide Template: see references/qualitative_protocols.md.
11. Focus Group Facilitation
Plan groups of 6-10 (4-6 for complex topics), 3-5 groups per segment until saturation, homogeneous within and heterogeneous across. Assign moderator and note-taker roles, prepare a neutral environment, and use funnel-approach facilitation to manage dominant and quiet voices.
Full Focus Group Design Checklist: see references/qualitative_protocols.md.
12. Cultural Adaptation of Instruments
Adapt instruments across languages/cultures using Brislin's (1970) back-translation cycle (forward translation → independent back-translation → reconciliation → cultural review → cognitive interviews → validation) and the 10-step ISPOR cross-cultural adaptation guidelines.
Full back-translation flow diagram and ISPOR step list: see references/qualitative_protocols.md.
Best Practices
Start with constructs, not questions. Define exactly what you are measuring before writing a single item. Each item should trace back to a specific construct or dimension.
Use existing validated instruments when possible. Do not reinvent the wheel. Search the literature for instruments with established psychometric properties.
Pilot everything. Every survey should go through cognitive interviews and a quantitative pilot before full deployment. There is no substitute for testing with your target population.
Keep it short. Every additional item increases dropout risk. Include only items you will actually analyze. A good survey is as short as possible and as long as necessary.
Design for your weakest respondent. Write at an appropriate reading level. Test on mobile devices. Consider accessibility (screen readers, color contrast). Provide translations if needed.
Randomize item order within sections. This reduces order effects and helps detect careless responding.
Include attention checks. Embed 1-2 instructed response items (e.g., "Please select 'Agree' for this item") to identify careless respondents.
Plan your analysis before collecting data. Every question should have a purpose in your analysis plan. If you cannot say how you will analyze an item, remove it.
Document everything. Keep a survey design log recording every decision: why items were added, removed, or revised; pilot test results; expert feedback.
Protect respondent data. Use anonymous links when possible; store data securely; minimize collection of identifiers; comply with IRB requirements.
Common Pitfalls
| Pitfall |
Why It Happens |
How to Avoid |
| Double-barreled questions |
Trying to be efficient; asking two things at once |
Split into separate items; one concept per item |
| Leading questions |
Researcher's hypothesis influences wording |
Have a colleague blind to your hypothesis review items |
| Response options that do not match the stem |
Copy-pasting from another survey |
Ensure stem and response scale are grammatically and logically matched |
| Too many open-ended questions |
Wanting rich data |
Limit to 2-3 open-ended items; save depth for interviews |
| No pilot testing |
Time pressure; overconfidence in item clarity |
Always pilot — even a quick cognitive interview with 3-5 people helps |
| Ignoring mobile respondents |
Designing on desktop |
Test on multiple devices; avoid matrix questions on mobile (they break) |
| Low response rate |
No follow-up plan; survey too long; no incentive |
Pre-notify; send reminders (3-4 contacts); shorten survey; offer incentive |
| Neglecting psychometric validation |
Assuming items are valid because they "look right" |
Run reliability and factor analysis; report results in your paper |
| Convenience sampling reported as representative |
Not understanding sampling limitations |
Be honest about sampling method in limitations section |
| Cultural insensitivity |
Assuming instruments transfer across cultures |
Use formal adaptation procedures (back-translation, cognitive interviews) |
References
- DeVellis, R. F., & Thorpe, C. T. (2022). Scale development: Theory and applications (5th ed.). Sage.
- Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley.
- Fowler, F. J. (2014). Survey research methods (5th ed.). Sage.
- Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Wiley.
- Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design. In P. V. Marsden & J. D. Wright (Eds.), Handbook of survey research (2nd ed., pp. 263-313). Emerald.
- Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research. Journal of Applied Psychology, 88(5), 879-903.
- Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage.
See also: references/survey-methodology.md for expanded methodology details.
1---2name: alterlab-survey-design3description: Comprehensive survey and instrument design assistant supporting questionnaire construction, Likert scale design, question types (open/closed/matrix), response bias mitigation, sampling strategies (probability/non-probability), pilot testing, instrument validation (Cronbach's alpha, factor analysis), online survey tools (Qualtrics, REDCap, Google Forms), interview protocol development, focus group facilitation, mixed-mode surveys, and cultural adaptation of instruments. Use when designing a survey or questionnaire, building Likert scales, planning a sampling strategy, pilot testing, validating an instrument (Cronbach's alpha, factor analysis), developing an interview protocol, improving response rates, or working in Qualtrics or REDCap. For analyzing interview/focus-group data use alterlab-qualitative-methods; for qual+quant integration alterlab-mixed-methods; for test selection/power analysis alterlab-statistical-analysis; for IRB/consent alterlab-research-ethics. Part of the AlterLab Academic Skills suite.4license: MIT5---67# Survey Design — Survey & Instrument Design Agent89A comprehensive survey and instrument design tool for faculty and researchers. Covers the full lifecycle of survey-based research: from construct definition and item writing through pilot testing, validation, deployment, and analysis of survey data.1011## Overview1213Survey research is one of the most widely used methods across social sciences, health sciences, education, and business. Despite its apparent simplicity, designing a valid and reliable survey instrument requires systematic attention to construct definition, item wording, response format, sampling, bias mitigation, and psychometric validation.1415This skill treats survey design as a scientific process, not an art. Every design decision should be justified and documented.1617## When to Use This Skill1819This skill should be used when:20- Designing a new survey or questionnaire from scratch21- Adapting an existing instrument for a new population or context22- Writing Likert-scale items or other structured response formats23- Developing interview protocols or focus group guides24- Planning sampling strategies for survey research25- Conducting pilot tests and cognitive interviews26- Validating instruments (reliability and validity analysis)27- Selecting online survey platforms (Qualtrics, REDCap, Google Forms)28- Improving response rates and reducing bias29- Conducting cultural adaptation and translation of instruments30- Teaching research methods courses that include survey design3132### Does NOT Trigger3334| Scenario | Use Instead |35|----------|-------------|36| Qualitative data analysis (coding, themes, focus group/interview analysis) | `alterlab-qualitative-methods` |37| Integrating qual + quant strands (convergent/sequential designs, joint displays) | `alterlab-mixed-methods` |38| Hypothesis-test selection, assumption checks, power analysis beyond validation | `alterlab-statistical-analysis` |39| Specialized social-science methods (Delphi, Q-methodology, QCA) | `alterlab-social-science-methods` |40| Writing the research paper | `alterlab-paper-writer` |41| Ethics/IRB applications, informed consent for survey research | `alterlab-research-ethics` |4243---4445## Core Capabilities4647### 1. Survey Design Process4849**The 10-Step Survey Design Framework:**5051```52Step 1: Define research objectives and constructs53 What do you want to measure? What are your research questions?54 │55Step 2: Review existing instruments56 Has someone already validated an instrument for your construct?57 │58Step 3: Define the target population and sampling frame59 Who will you survey? How will you reach them?60 │61Step 4: Choose survey mode62 Online, paper, phone, in-person, mixed-mode?63 │64Step 5: Write items and response options65 Craft questions that are clear, unambiguous, and aligned to constructs66 │67Step 6: Design survey structure and flow68 Organize sections, add skip logic, manage survey length69 │70Step 7: Expert review71 Subject matter experts and methodologists evaluate the instrument72 │73Step 8: Cognitive interviews and pilot testing74 Test with a small sample from the target population75 │76Step 9: Psychometric validation77 Reliability analysis, factor analysis, validity assessment78 │79Step 10: Deploy, monitor, and analyze80 Launch survey, track response rates, clean and analyze data81```8283### 2. Construct Definition and Operationalization8485Before writing a single item, define what you are measuring. Map each construct to its dimensions, indicators, distinct-but-related constructs, and a nomological network so every item traces back to something specific.8687Full **Construct Mapping Template**: see `references/item_writing_and_scales.md`.8889### 3. Item Writing9091#### Question Types and When to Use Them9293| Type | Format | Best For |94|------|--------|----------|95| Closed-ended (single choice) | Radio buttons | Mutually exclusive categories |96| Closed-ended (multiple choice) | Checkboxes | Non-mutually exclusive categories |97| Likert scale | Rating scale | Attitudes, perceptions, frequency |98| Semantic differential | Bipolar scale | Evaluative judgments |99| Ranking | Drag-and-drop or numbered | Forced prioritization |100| Matrix/Grid | Likert items in table | Multiple items with same response scale |101| Open-ended | Text box | Exploratory, rich responses |102| Numeric | Number input | Precise quantities |103| Visual analog scale (VAS) | Slider | Continuous measurement |104105#### Likert Scale — Number of Points106107| Points | Trade-off | Use When |108|--------|-----------|----------|109| 4-point | Forces a choice (no midpoint) | Avoid social desirability midpoint clustering |110| 5-point | Most common, well-understood; central-tendency bias | Standard attitudinal measurement |111| 6-point | Forced choice with more granularity | Force direction with more options |112| 7-point | Greater discrimination; better for factor analysis | Established psychometric instruments |113114**Item-writing essentials — DO:** simple/clear language; one concept per item (no double-barreled); specific time frames; match scale to stem; pilot with the target population; over-generate items. **DO NOT:** leading/loaded language; double negatives; assume knowledge; use absolutes; write overly long items; ask about hypotheticals when you mean actual behavior.115116Full question-type examples, Likert labeling schemes, the complete DO / DO NOT rules, and worked before/after item revisions: see `references/item_writing_and_scales.md`.117118### 4. Survey Structure and Flow119120Organize the instrument as: welcome + consent → screening → main content grouped by construct (easy questions first, sensitive items mid-survey) → demographics at the end → thank-you/debrief. Use skip logic to hide irrelevant questions and route ineligible respondents.121122Full **Survey Structure Template** and **Skip Logic Design** examples: see `references/item_writing_and_scales.md`.123124### 5. Sampling Strategies125126**Probability Sampling** (every member has a known, non-zero chance of selection; generalizable):127128| Method | How It Works | Trade-off |129|--------|-------------|-----------|130| Simple random | Select randomly from complete list | Unbiased, but needs a complete sampling frame |131| Systematic | Select every kth element | Easy, but periodicity risk if list has a pattern |132| Stratified | Random sample within population strata | Ensures subgroup representation; needs population knowledge |133| Cluster | Randomly select clusters, then sample within | Practical without individual list; higher sampling error |134| Multi-stage | Combine methods (cluster then stratified) | Flexible for large populations; complex to analyze |135136**Non-Probability Sampling** (no representativeness guarantee):137138| Method | How It Works | Trade-off |139|--------|-------------|-----------|140| Convenience | Recruit whoever is available | Fast/cheap, but strong bias |141| Purposive | Select on specific criteria | Targets relevant subgroups; researcher bias |142| Snowball | Participants recruit others | Reaches hidden populations; biased toward the connected |143| Quota | Convenience sample within subgroup quotas | Ensures diversity; not truly random within quotas |144145Size the sample with the proportion formula `n = (Z² × p × (1-p)) / E²` for descriptive surveys (adjusting for finite population and expected response rate), or a power analysis for comparative surveys. Worked sample-size formulas and the Python two-group power-analysis helper: see `references/sampling_and_power.md`.146147### 6. Response Bias Mitigation148149| Bias Type | Definition | Mitigation Strategies |150|-----------|-----------|----------------------|151| Social desirability | Respondents answer in ways they believe are socially acceptable | Anonymous data collection; indirect questioning; validated social desirability scales (e.g., Marlowe-Crowne) |152| Acquiescence | Tendency to agree with statements regardless of content | Mix positively and negatively worded items; use forced-choice formats |153| Central tendency | Tendency to select middle response options | Use even-point scales (no midpoint); provide behavioral anchors |154| Extreme responding | Tendency to select extreme endpoints | Use more response options (7-point); provide clear anchor descriptions |155| Order effects | Earlier questions influence responses to later questions | Randomize item order within sections; counterbalance across respondents |156| Nonresponse bias | Systematic differences between responders and non-responders | Follow-up reminders; analyze early vs. late responders; compare demographics to population |157| Recall bias | Inaccurate recall of past events | Use shorter recall periods; provide memory aids; use event-specific prompts |158| Common method bias | Inflated correlations due to same measurement method | Use different measurement methods; temporal separation; marker variables |159160### 7. Pilot Testing161162Run a **three-phase pilot** before full deployment: (1) expert review for content/face validity (CVI thresholds: Item-CVI ≥ 0.78, Scale-CVI/Ave ≥ 0.90); (2) cognitive interviews (n = 5-10) using think-aloud and probing questions; (3) a quantitative pilot (n = 30-50) assessing completion, missing data, distributions, internal consistency, and item-total correlations.163164Full phase-by-phase protocol with probe scripts and the quantitative-pilot checklist: see `references/pilot_and_validation.md`.165166### 8. Instrument Validation167168Assess **reliability** (Cronbach's alpha per subscale, corrected item-total correlations — flag items < 0.30) and **validity** across the evidence types below. Use exploratory factor analysis (Bartlett's test, KMO, eigenvalues, rotated loadings) to check internal structure.169170| Type | Question | Method |171|------|----------|--------|172| Content validity | Do items cover the construct adequately? | Expert review, CVI calculation |173| Face validity | Do items appear to measure the construct? | Target population review |174| Construct validity | Does it measure the theoretical construct? | Factor analysis (EFA/CFA) |175| Convergent validity | Does it correlate with similar measures? | Correlation with established instruments (r > 0.50) |176| Discriminant validity | Is it distinct from different constructs? | Low correlation with unrelated measures (r < 0.30) |177| Criterion (concurrent) | Does it correlate with a current criterion? | Correlation with gold standard, measured simultaneously |178| Criterion (predictive) | Does it predict a future outcome? | Correlation with criterion measured later |179| Known-groups | Can it distinguish groups known to differ? | Compare scores between groups that should differ |180181Runnable Python for Cronbach's alpha, item-total correlations, EFA, and the three-phase pilot protocol: see `references/pilot_and_validation.md`. For CFA fit indices, measurement invariance, and the broader psychometric framework, see `references/survey-methodology.md`.182183### 9. Online Survey Platform Comparison184185| Feature | Qualtrics | REDCap | Google Forms | LimeSurvey |186|---------|-----------|--------|--------------|------------|187| Cost | Institutional license (expensive) | Free for institutions | Free | Free (open source) |188| Skip logic | Advanced | Advanced | Basic | Advanced |189| Randomization | Yes (items, blocks) | Limited | No | Yes |190| Piping | Yes | Yes | No | Yes |191| Offline data collection | Yes (app) | Yes (app) | No | Yes |192| HIPAA compliant | Yes (BAA available) | Yes (designed for it) | No | Self-hosted: yes |193| API access | Yes | Yes | Limited | Yes |194| Data export | CSV, SPSS, Excel | CSV, Excel, SPSS, SAS, R, Stata | CSV, Excel | CSV, Excel, SPSS, R |195| Multi-language | Yes | Yes | Manual | Yes |196| Panel integration | Yes (Prolific, MTurk) | No | No | Limited |197| Best for | Complex academic surveys | Clinical and health research | Simple surveys, course evaluations | Budget-conscious complex surveys |198199### 10. Interview Protocol Development200201Build semi-structured interview guides with a scripted opening/consent, a warm-up question, main-question blocks organized by construct (each with probes), a closing catch-all, and a post-interview field-notes routine.202203Full **Semi-Structured Interview Guide Template**: see `references/qualitative_protocols.md`.204205### 11. Focus Group Facilitation206207Plan groups of 6-10 (4-6 for complex topics), 3-5 groups per segment until saturation, homogeneous within and heterogeneous across. Assign moderator and note-taker roles, prepare a neutral environment, and use funnel-approach facilitation to manage dominant and quiet voices.208209Full **Focus Group Design Checklist**: see `references/qualitative_protocols.md`.210211### 12. Cultural Adaptation of Instruments212213Adapt instruments across languages/cultures using Brislin's (1970) back-translation cycle (forward translation → independent back-translation → reconciliation → cultural review → cognitive interviews → validation) and the 10-step ISPOR cross-cultural adaptation guidelines.214215Full back-translation flow diagram and ISPOR step list: see `references/qualitative_protocols.md`.216217---218219## Best Practices2202211. **Start with constructs, not questions.** Define exactly what you are measuring before writing a single item. Each item should trace back to a specific construct or dimension.2222232. **Use existing validated instruments when possible.** Do not reinvent the wheel. Search the literature for instruments with established psychometric properties.2242253. **Pilot everything.** Every survey should go through cognitive interviews and a quantitative pilot before full deployment. There is no substitute for testing with your target population.2262274. **Keep it short.** Every additional item increases dropout risk. Include only items you will actually analyze. A good survey is as short as possible and as long as necessary.2282295. **Design for your weakest respondent.** Write at an appropriate reading level. Test on mobile devices. Consider accessibility (screen readers, color contrast). Provide translations if needed.2302316. **Randomize item order within sections.** This reduces order effects and helps detect careless responding.2322337. **Include attention checks.** Embed 1-2 instructed response items (e.g., "Please select 'Agree' for this item") to identify careless respondents.2342358. **Plan your analysis before collecting data.** Every question should have a purpose in your analysis plan. If you cannot say how you will analyze an item, remove it.2362379. **Document everything.** Keep a survey design log recording every decision: why items were added, removed, or revised; pilot test results; expert feedback.23823910. **Protect respondent data.** Use anonymous links when possible; store data securely; minimize collection of identifiers; comply with IRB requirements.240241---242243## Common Pitfalls244245| Pitfall | Why It Happens | How to Avoid |246|---------|---------------|--------------|247| Double-barreled questions | Trying to be efficient; asking two things at once | Split into separate items; one concept per item |248| Leading questions | Researcher's hypothesis influences wording | Have a colleague blind to your hypothesis review items |249| Response options that do not match the stem | Copy-pasting from another survey | Ensure stem and response scale are grammatically and logically matched |250| Too many open-ended questions | Wanting rich data | Limit to 2-3 open-ended items; save depth for interviews |251| No pilot testing | Time pressure; overconfidence in item clarity | Always pilot — even a quick cognitive interview with 3-5 people helps |252| Ignoring mobile respondents | Designing on desktop | Test on multiple devices; avoid matrix questions on mobile (they break) |253| Low response rate | No follow-up plan; survey too long; no incentive | Pre-notify; send reminders (3-4 contacts); shorten survey; offer incentive |254| Neglecting psychometric validation | Assuming items are valid because they "look right" | Run reliability and factor analysis; report results in your paper |255| Convenience sampling reported as representative | Not understanding sampling limitations | Be honest about sampling method in limitations section |256| Cultural insensitivity | Assuming instruments transfer across cultures | Use formal adaptation procedures (back-translation, cognitive interviews) |257258---259260## References261262- DeVellis, R. F., & Thorpe, C. T. (2022). *Scale development: Theory and applications* (5th ed.). Sage.263- Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). *Internet, phone, mail, and mixed-mode surveys: The tailored design method* (4th ed.). Wiley.264- Fowler, F. J. (2014). *Survey research methods* (5th ed.). Sage.265- Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). *Survey methodology* (2nd ed.). Wiley.266- Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design. In P. V. Marsden & J. D. Wright (Eds.), *Handbook of survey research* (2nd ed., pp. 263-313). Emerald.267- Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research. *Journal of Applied Psychology*, 88(5), 879-903.268- Willis, G. B. (2005). *Cognitive interviewing: A tool for improving questionnaire design*. Sage.269270See also: `references/survey-methodology.md` for expanded methodology details.
AlterLab-IEU