Usability Testing
When to Use
- Use this skill when: Planning, executing, or summarizing usability tests (moderated or unmoderated) for design prototypes, websites, or mobile apps.
- Trigger keywords: "usability test", "user testing", "moderated session", "unmoderated study", "SUS score", "task success rate", "think-aloud", "UX research plan", "prototype validation".
- Route to Research-Ideation (Summarize Interview): If reviewing open-ended discovery interviews, user habits, or workflow conversations that do not involve active prototype interaction.
- Route to Research-Ideation (Product Feedback Synthesizer): If analyzing multi-channel product tickets, NPS data, or App Store reviews.
Prerequisites
- A functional interactive prototype (e.g., Figma link) or staging site URL.
- A usability test plan detailing target tasks, success metrics, and user profiles.
- Consent forms and NDAs prepared for all participants before testing unreleased designs or recording sessions.
- A 20% recruit buffer to account for no-shows.
Procedure
1. Define Objectives and Participant Profile
- Draft a usability test plan using the template below. Specify 1–3 primary objectives.
- Identify user segments. Plan for 5–8 participants per distinct segment for formative (qualitative) studies, or 20–40+ participants for summative (quantitative) validation.
- Write screener criteria that match the target demographic (e.g., "Must have completed an online purchase in the last 30 days").
Usability Test Plan Template:
# Usability Test Plan: [Project Name]
## Objectives
- [Objective 1: e.g., Validate checkout funnel drop-off points]
- [Objective 2: e.g., Evaluate discoverability of the filter panel]
## Participant Profile & Recruitment
- **Target Count**: e.g., 5-8 participants per user segment
- **Key Demographics**: e.g., Online shoppers, active mobile users
- **Screener Criteria**: e.g., Must have completed an online purchase in the last 30 days
## Methodology
- **Format**: [Remote / In-Person]
- **Type**: [Moderated / Unmoderated]
- **Tools**: [e.g., Maze, Figma, Lookback]
## Tasks & Scenarios
1. **Scenario 1: [Context]**
- *Task*: [Action required]
- *Success Criteria*: [Expected end state]
- *Follow-up questions*: [Probe questions]
2. **Scenario 2: [Context]**
- *Task*: [Action required]
- *Success Criteria*: [Expected end state]
2. Choose Methodology
| Method |
Best For |
Participant Count |
| Moderated |
Early-stage prototypes, complex B2B workflows, concept evaluation |
5–8 per segment |
| Unmoderated |
Finalized consumer flows (checkout, login), large-scale quantitative metrics |
20–40+ |
| Formative (Qualitative) |
Finding why interfaces fail, understanding mental models |
5–8 per segment |
| Summative (Quantitative) |
Validating how much design updates improve TSR, SUS, ToT |
20–40+ |
3. Prepare Tasks and Scenarios
- Write each task as a scenario with context, not a directive command. Example: "You need to buy a gift for a friend under $50. Show me how you'd do that on this site."
- Define explicit success criteria for each task (the expected end state).
- Prepare neutral follow-up probes. Use scripts like: "What did you expect to happen when you clicked that?"
- Prepare a post-test SUS questionnaire (10 standardized items, scored 0–100).
4. Set Up Testing Platform
- Remote unmoderated: Use Maze, UXtweak, or PlaybookUX. These support native Figma prototype syncing and heatmaps.
- Remote moderated: Use Lookback or Zoom with screen sharing.
- AI-moderated: Platforms like Koji and CleverX deploy autonomous AI moderators to probe users, bridging unmoderated scale with moderated depth.
- Ensure prototype links support mobile web previews and offline fallback layouts for users on low-bandwidth networks.
5. Facilitate Sessions (Moderated)
- Establish comfort: Begin by stating: "We are testing the product, not you. There are no right or wrong answers."
- Encourage think-aloud: Remind the participant to verbalize thoughts, doubts, and emotions as they navigate.
- Avoid leading questions: Never ask "Was that easy?" Ask instead: "How was that experience for you?"
- Avoid intervention: If a participant gets stuck, resist the urge to help immediately. Observe how they attempt to resolve the issue.
- Record timestamps for every observed blocker, error, or moment of confusion.
6. Collect Metrics
Track the following for each task:
- Task Success Rate (TSR): Percentage of participants who successfully complete the task.
- Time-on-Task (ToT): Average duration required to complete the task.
- Error Rate: Average number of errors or incorrect clicks per participant per task.
- System Usability Scale (SUS): Standardized 10-item questionnaire yielding a score from 0 to 100.
- Critical blockers: Any issue that prevents task completion entirely.
7. Analyze and Report
- Categorize every observed blocker by severity: Critical, Major, or Minor.
- Reference timestamped recording clips for every documented issue.
- Verify whether keyboard-only or screen-reader users encountered blockers (WCAG 2.2 accessibility checks).
- Produce the output report conforming to the JSON schema below.
Usability Metrics JSON Schema:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "UsabilityTestingReport",
"type": "object",
"properties": {
"testId": { "type": "string" },
"methodology": { "type": "string", "enum": ["moderated", "unmoderated"] },
"totalParticipants": { "type": "integer" },
"taskMetrics": {
"type": "array",
"items": {
"type": "object",
"properties": {
"taskId": { "type": "string" },
"description": { "type": "string" },
"successRatePercent": { "type": "number", "minimum": 0, "maximum": 100 },
"averageTimeSeconds": { "type": "number" },
"criticalBlockers": {
"type": "array",
"items": { "type": "string" }
}
},
"required": ["taskId", "description", "successRatePercent", "averageTimeSeconds", "criticalBlockers"]
}
},
"systemUsabilityScaleScore": { "type": "number", "minimum": 0, "maximum": 100 }
},
"required": ["testId", "methodology", "totalParticipants", "taskMetrics"]
}
Pitfalls
- Leading the participant: Providing hints or explaining how a feature works during a testing session invalidates behavioral data. Never intervene unless the participant is completely blocked and has given up.
- Over-modifying prototypes mid-study: Making changes to Figma designs between sessions invalidates comparisons across participants. Lock the prototype before the first session.
- Focusing on opinions over behavior: Prioritizing participant statements like "I like this blue" over their actual interaction behaviors (e.g., failing to click the blue button). Always weight observed action above self-reported preference.
- Insufficient sample size: Running formative studies with fewer than 5 users per segment risks missing critical issues. Running quantitative validation with fewer than 20 participants produces statistically inconclusive results.
- Recording sensitive data: Do not record password input screens or display personal account details in shared reports. Protect participant privacy.
- Skipping accessibility checks: Modern usability testing protocols require verifying WCAG 2.2 accessibility parameters — interactive components must support screen reader cues and keyboard focus mapping.
- No backup plan for prototype failures: If a Figma link fails or runs slowly during a session, switch immediately to a backup static layout deck and document the tech issue.
Verification
Before finalizing any usability test report, confirm:
- Every blocker is categorized: No unresolved blockers remain. Each is labeled Critical, Major, or Minor.
- Video grounding exists: Every documented issue references a timestamped recording clip.
- Accessibility verified: Report states whether keyboard-only or screen-reader users encountered blockers.
- SUS score computed: If summative, the System Usability Scale score (0–100) is present and calculated per the standard methodology.
- JSON schema conformance: The output report validates against the
UsabilityTestingReport JSON schema above.
- Consent and NDA on file: All participants signed consent forms before the session began.
Quick validation checklist:
Related Skills
- research-ideation: Use for open-ended discovery interviews, workflow conversations, or multi-channel product feedback synthesis that does not involve active prototype interaction.
- accessibility-audit: Use for systematic WCAG 2.2 conformance testing separate from user-session-based usability testing.
Source Anchors
Changelog
- 2026-05-30: Modernized skill file created. Updated frontmatter, corrected second-person phrasing, removed boilerplate, and added Maze, UXtweak, and AI-moderation details.
- 2026-05-31: Production-grade rewrite with progressive disclosure structure, explicit procedure steps, JSON schema, and verification checklist.
1---
2name: usability-testing
3description: "Plans and runs moderated or unmoderated usability sessions: 5-8 formative or 20-40+ summative recruits, think-aloud, TSR/ToT/SUS, and a severity-coded JSON report. Use when the user asks for a usability test, Maze or Lookback session, SUS score, or prototype validation with participants. Never a discovery-interview synthesizer and not a WCAG-only accessibility audit."
4version: 1.0.1
5domain: ui-ux
6risk: safe
7last_verified: '2026-05-30'
8self_updating: true
9metadata:
10 source: https://skillmd.com/skills/kayforkind/usability-testing
11---
12
13# Usability Testing
14
15## When to Use
16
17- **Use this skill when**: Planning, executing, or summarizing usability tests (moderated or unmoderated) for design prototypes, websites, or mobile apps.
18- **Trigger keywords**: "usability test", "user testing", "moderated session", "unmoderated study", "SUS score", "task success rate", "think-aloud", "UX research plan", "prototype validation".
19- **Route to Research-Ideation (Summarize Interview)**: If reviewing open-ended discovery interviews, user habits, or workflow conversations that do not involve active prototype interaction.
20- **Route to Research-Ideation (Product Feedback Synthesizer)**: If analyzing multi-channel product tickets, NPS data, or App Store reviews.
21
22## Prerequisites
23
24- A functional interactive prototype (e.g., Figma link) or staging site URL.
25- A usability test plan detailing target tasks, success metrics, and user profiles.
26- Consent forms and NDAs prepared for all participants before testing unreleased designs or recording sessions.
27- A 20% recruit buffer to account for no-shows.
28
29## Procedure
30
31### 1. Define Objectives and Participant Profile
32
331. Draft a usability test plan using the template below. Specify 1–3 primary objectives.
342. Identify user segments. Plan for **5–8 participants per distinct segment** for formative (qualitative) studies, or **20–40+ participants** for summative (quantitative) validation.
353. Write screener criteria that match the target demographic (e.g., "Must have completed an online purchase in the last 30 days").
36
37**Usability Test Plan Template:**
38
39```markdown
40# Usability Test Plan: [Project Name]
41
42## Objectives
43- [Objective 1: e.g., Validate checkout funnel drop-off points]
44- [Objective 2: e.g., Evaluate discoverability of the filter panel]
45
46## Participant Profile & Recruitment
47- **Target Count**: e.g., 5-8 participants per user segment
48- **Key Demographics**: e.g., Online shoppers, active mobile users
49- **Screener Criteria**: e.g., Must have completed an online purchase in the last 30 days
50
51## Methodology
52- **Format**: [Remote / In-Person]
53- **Type**: [Moderated / Unmoderated]
54- **Tools**: [e.g., Maze, Figma, Lookback]
55
56## Tasks & Scenarios
571. **Scenario 1: [Context]**
58 - *Task*: [Action required]
59 - *Success Criteria*: [Expected end state]
60 - *Follow-up questions*: [Probe questions]
61
622. **Scenario 2: [Context]**
63 - *Task*: [Action required]
64 - *Success Criteria*: [Expected end state]
65```
66
67### 2. Choose Methodology
68
69| Method | Best For | Participant Count |
70|---|---|---|
71| Moderated | Early-stage prototypes, complex B2B workflows, concept evaluation | 5–8 per segment |
72| Unmoderated | Finalized consumer flows (checkout, login), large-scale quantitative metrics | 20–40+ |
73| Formative (Qualitative) | Finding *why* interfaces fail, understanding mental models | 5–8 per segment |
74| Summative (Quantitative) | Validating *how much* design updates improve TSR, SUS, ToT | 20–40+ |
75
76### 3. Prepare Tasks and Scenarios
77
781. Write each task as a scenario with context, not a directive command. Example: *"You need to buy a gift for a friend under $50. Show me how you'd do that on this site."*
792. Define explicit success criteria for each task (the expected end state).
803. Prepare neutral follow-up probes. Use scripts like: *"What did you expect to happen when you clicked that?"*
814. Prepare a post-test SUS questionnaire (10 standardized items, scored 0–100).
82
83### 4. Set Up Testing Platform
84
85- **Remote unmoderated**: Use [Maze](https://www.maze.co), [UXtweak](https://www.uxtweak.com), or [PlaybookUX](https://www.playbookux.com). These support native Figma prototype syncing and heatmaps.
86- **Remote moderated**: Use Lookback or Zoom with screen sharing.
87- **AI-moderated**: Platforms like **Koji** and **CleverX** deploy autonomous AI moderators to probe users, bridging unmoderated scale with moderated depth.
88- Ensure prototype links support mobile web previews and offline fallback layouts for users on low-bandwidth networks.
89
90### 5. Facilitate Sessions (Moderated)
91
921. **Establish comfort**: Begin by stating: *"We are testing the product, not you. There are no right or wrong answers."*
932. **Encourage think-aloud**: Remind the participant to verbalize thoughts, doubts, and emotions as they navigate.
943. **Avoid leading questions**: Never ask *"Was that easy?"* Ask instead: *"How was that experience for you?"*
954. **Avoid intervention**: If a participant gets stuck, resist the urge to help immediately. Observe how they attempt to resolve the issue.
965. Record timestamps for every observed blocker, error, or moment of confusion.
97
98### 6. Collect Metrics
99
100Track the following for each task:
101
102- **Task Success Rate (TSR)**: Percentage of participants who successfully complete the task.
103- **Time-on-Task (ToT)**: Average duration required to complete the task.
104- **Error Rate**: Average number of errors or incorrect clicks per participant per task.
105- **System Usability Scale (SUS)**: Standardized 10-item questionnaire yielding a score from 0 to 100.
106- **Critical blockers**: Any issue that prevents task completion entirely.
107
108### 7. Analyze and Report
109
1101. Categorize every observed blocker by severity: **Critical**, **Major**, or **Minor**.
1112. Reference timestamped recording clips for every documented issue.
1123. Verify whether keyboard-only or screen-reader users encountered blockers (WCAG 2.2 accessibility checks).
1134. Produce the output report conforming to the JSON schema below.
114
115**Usability Metrics JSON Schema:**
116
117```json
118{
119 "$schema": "http://json-schema.org/draft-07/schema#",
120 "title": "UsabilityTestingReport",
121 "type": "object",
122 "properties": {
123 "testId": { "type": "string" },
124 "methodology": { "type": "string", "enum": ["moderated", "unmoderated"] },
125 "totalParticipants": { "type": "integer" },
126 "taskMetrics": {
127 "type": "array",
128 "items": {
129 "type": "object",
130 "properties": {
131 "taskId": { "type": "string" },
132 "description": { "type": "string" },
133 "successRatePercent": { "type": "number", "minimum": 0, "maximum": 100 },
134 "averageTimeSeconds": { "type": "number" },
135 "criticalBlockers": {
136 "type": "array",
137 "items": { "type": "string" }
138 }
139 },
140 "required": ["taskId", "description", "successRatePercent", "averageTimeSeconds", "criticalBlockers"]
141 }
142 },
143 "systemUsabilityScaleScore": { "type": "number", "minimum": 0, "maximum": 100 }
144 },
145 "required": ["testId", "methodology", "totalParticipants", "taskMetrics"]
146}
147```
148
149## Pitfalls
150
151- **Leading the participant**: Providing hints or explaining how a feature works during a testing session invalidates behavioral data. Never intervene unless the participant is completely blocked and has given up.
152- **Over-modifying prototypes mid-study**: Making changes to Figma designs between sessions invalidates comparisons across participants. Lock the prototype before the first session.
153- **Focusing on opinions over behavior**: Prioritizing participant statements like *"I like this blue"* over their actual interaction behaviors (e.g., failing to click the blue button). Always weight observed action above self-reported preference.
154- **Insufficient sample size**: Running formative studies with fewer than 5 users per segment risks missing critical issues. Running quantitative validation with fewer than 20 participants produces statistically inconclusive results.
155- **Recording sensitive data**: Do not record password input screens or display personal account details in shared reports. Protect participant privacy.
156- **Skipping accessibility checks**: Modern usability testing protocols require verifying WCAG 2.2 accessibility parameters — interactive components must support screen reader cues and keyboard focus mapping.
157- **No backup plan for prototype failures**: If a Figma link fails or runs slowly during a session, switch immediately to a backup static layout deck and document the tech issue.
158
159## Verification
160
161Before finalizing any usability test report, confirm:
162
1631. **Every blocker is categorized**: No unresolved blockers remain. Each is labeled Critical, Major, or Minor.
1642. **Video grounding exists**: Every documented issue references a timestamped recording clip.
1653. **Accessibility verified**: Report states whether keyboard-only or screen-reader users encountered blockers.
1664. **SUS score computed**: If summative, the System Usability Scale score (0–100) is present and calculated per the standard methodology.
1675. **JSON schema conformance**: The output report validates against the `UsabilityTestingReport` JSON schema above.
1686. **Consent and NDA on file**: All participants signed consent forms before the session began.
169
170**Quick validation checklist:**
171
172- [ ] Test plan defines objectives, participant profile, and success criteria
173- [ ] Prototype link tested and functional before first session
174- [ ] 5–8 users per segment (formative) or 20–40+ (summative) recruited
175- [ ] 20% no-show buffer applied to recruitment
176- [ ] Facilitator script uses neutral, non-leading language
177- [ ] All sessions recorded with timestamps
178- [ ] Blockers categorized by severity
179- [ ] Accessibility checks performed and documented
180- [ ] Report conforms to JSON output schema
181
182## Related Skills
183
184- **research-ideation**: Use for open-ended discovery interviews, workflow conversations, or multi-channel product feedback synthesis that does not involve active prototype interaction.
185- **accessibility-audit**: Use for systematic WCAG 2.2 conformance testing separate from user-session-based usability testing.
186
187## Source Anchors
188
189- [Nielsen Norman Group Usability Testing 101](https://www.nngroup.com/articles/usability-testing-101/)
190- [W3C WCAG 2.2](https://www.w3.org/TR/WCAG22/)
191- [Apple Human Interface Guidelines](https://developer.apple.com/design/human-interface-guidelines/)
192- [Material Design 3](https://m3.material.io/)
193- [Maze Usability Testing Suite](https://maze.co/)
194- [UXtweak Remote Testing Platform](https://www.uxtweak.com/)
195- [PlaybookUX Research Platform](https://www.playbookux.com/)
196
197## Changelog
198
199- **2026-05-30**: Modernized skill file created. Updated frontmatter, corrected second-person phrasing, removed boilerplate, and added Maze, UXtweak, and AI-moderation details.
200- **2026-05-31**: Production-grade rewrite with progressive disclosure structure, explicit procedure steps, JSON schema, and verification checklist.