xrnavigation Accessibility Testing Strategy
"Automated testing with axe-core identifies 57% of digital accessibility issues by volume." — Deque, Automated Testing Study (2023)
"Automated scans reliably flag only 13% of WCAG 2.1 AA success criteria." — Accessible.org
Automated testing is necessary but insufficient. A complete accessibility testing strategy uses three tiers, each catching different categories of issues.
1. Three Testing Tiers
Tier 1: Automated Testing (~57% of issues by volume)
Tools: axe-core, Lighthouse, WAVE, HTML validators When: Every build / CI pipeline
Catches well:
- Color contrast ratios below thresholds (WCAG 1.4.3)
- Missing
altattributes, form labels, pagelang(Deque, 2023) - Invalid ARIA: duplicate IDs, broken references, invalid roles
- Structural issues: missing bypass blocks, empty/skipped headings
- Missing page title, missing language declaration
Key stat: Seven WCAG criteria account for over 80% of all issues found, with contrast alone at ~30%. (Deque, 2023)
Tier 2: Keyboard Testing
Tools: None required beyond a keyboard When: Every feature / PR
Catches issues automation misses:
- Focus visibility and indicator quality (WCAG 2.4.7)
- Tab order matching visual/reading order (WCAG 2.4.3)
- Keyboard traps (WCAG 2.1.2)
- Interactive element operability (WCAG 2.1.1)
- Skip links and modal focus management
See: references/keyboard-testing-checklist.md
Tier 3: Screen Reader Testing
Tools: NVDA + Chrome (Windows), VoiceOver + Safari (macOS/iOS), TalkBack + Chrome (Android) When: New components, significant UI changes, pre-release
Catches issues that require hearing what AT communicates:
- Reading order and meaningful sequence (WCAG 1.3.2)
- Live region announcements (WCAG 4.1.3)
- Name, role, value correctness (WCAG 4.1.2)
- Info and relationships communication (WCAG 1.3.1)
See: references/screen-reader-commands.md
2. What Automated Testing Catches vs Misses
Coverage Breakdown (WCAG 2.1 AA, 55 criteria)
| Detection Level | Criteria Count | Percentage |
|---|---|---|
| Mostly accurate automated detection | 7 of 55 | 13% |
| Partially detectable (flags, needs human review) | 25 of 55 | 45% |
| Cannot be detected by automation | 23 of 55 | 42% |
Source: Accessible.org
What Automation Cannot Catch
| Category | Why Automation Fails |
|---|---|
| Alt text quality | Detects missing alt but not whether text is meaningful |
| Reading order | DOM vs visual order requires human judgment |
| Keyboard operability | Dynamic interaction sequences can't be fully automated |
| Focus management | Context-dependent behavior in SPAs, modals |
| Meaningful labels | "Click here" is technically present but not descriptive |
| Error message quality | Detects presence but not helpfulness |
| Cognitive load | Information organization, plain language |
| Live region behavior | Timing and appropriateness of announcements |
Source: Deque, Manual vs Automated Testing
3. Keyboard Testing Summary
Disconnect the mouse. Start from the browser address bar.
| Check | What to Verify |
|---|---|
| Tab order | Tab moves through all interactive elements in logical order |
| Focus visibility | Every focused element has a visible indicator (3:1 contrast min) |
| Operability | Links: Enter. Buttons: Enter/Space. Checkboxes: Space. Radios: Arrows |
| No traps | Focus never gets stuck; Tab/Shift+Tab/Escape always works |
| Modals | Focus enters modal, cycles within it, Escape closes, focus returns |
| Skip links | First focusable element, moves focus past navigation |
Standard keys:
| Key | Expected Action |
|---|---|
| Tab / Shift+Tab | Move forward/backward through interactive elements |
| Enter | Activate link or button |
| Space | Activate button, toggle checkbox, open select |
| Arrow keys | Navigate within widget (radio group, menu, tabs, listbox) |
| Escape | Close modal, menu, or dialog |
| Home / End | Jump to first/last item in list or menu |
Sources: WebAIM Keyboard Accessibility, UW Keyboard Checklist, W3C Easy Checks
Full checklist: references/keyboard-testing-checklist.md
4. Screen Reader Testing Basics
Which SR + Browser Pairings to Test
Test in priority order based on market share (WebAIM Survey #10, 2024):
| Priority | Combination | Coverage | Cost |
|---|---|---|---|
| 1 | NVDA + Chrome (Windows) | 21.3% of users | Free |
| 2 | VoiceOver + Safari (macOS) | 7.0% + all iOS | Built-in |
| 3 | JAWS + Chrome (Windows) | 24.7% of users | License required |
| 4 | TalkBack + Chrome (Android) | 27.6% of mobile | Built-in |
Desktop Market Share
| Screen Reader | Share |
|---|---|
| JAWS | 40.5% |
| NVDA | 37.7% |
| VoiceOver | 9.7% |
What to Listen For
When testing with a screen reader, verify:
- Element identification — role announced correctly ("button", "link", "heading level 2")
- Accessible name — label is meaningful and matches visible text (WCAG 2.5.3)
- State changes — expanded/collapsed, checked/unchecked, selected announced on change
- Reading order — content reads in meaningful sequence
- Live regions — dynamic updates announced without requiring navigation
- Heading structure — headings navigable (H key in NVDA), hierarchy logical
Full command reference: references/screen-reader-commands.md
5. WCAG Criteria Requiring Human Judgment
These criteria cannot be verified by automation — they require understanding of meaning, context, and quality.
| Criterion | Why Human Judgment Needed |
|---|---|
| 1.1.1 Non-text Content (quality) | Can detect missing alt but not accuracy |
| 1.2.1–1.2.5 Media alternatives | Cannot verify transcript/caption accuracy |
| 1.3.2 Meaningful Sequence | Cannot determine if reading order preserves meaning |
| 1.3.3 Sensory Characteristics | Cannot detect reliance on shape/color/location alone |
| 2.4.5 Multiple Ways | Cannot assess navigation method availability across site |
| 2.4.6 Headings and Labels (quality) | Detects headings, not whether they are descriptive |
| 3.1.2 Language of Parts | Cannot determine where language changes occur |
| 3.2.3 Consistent Navigation | Cannot compare consistency across pages |
| 3.2.4 Consistent Identification | Cannot verify consistent naming across pages |
| 3.3.3 Error Suggestion | Cannot evaluate if error messages are helpful |
| 3.3.4 Error Prevention | Cannot verify confirmation/review for critical transactions |
Partially Detectable (flagged but need human confirmation)
| Criterion | Automation Flags | Humans Must Verify |
|---|---|---|
| 1.3.1 Info and Relationships | Missing semantic markup | Whether markup conveys right relationships |
| 1.4.5 Images of Text | Presence of images | Whether images contain text that should be HTML |
| 2.4.4 Link Purpose | Link text exists | Whether text is meaningful in context |
| 4.1.2 Name, Role, Value | ARIA attributes present | Whether names are accurate and useful |
Source: Accessible.org
6. Writing Acceptance Criteria
Per-Story Template
Add to each user story:
Acceptance Criteria:
- [ ] All interactive elements reachable and operable by keyboard
- [ ] Focus order matches visual/logical order
- [ ] Screen reader announces [element] as [expected announcement]
- [ ] Color contrast meets WCAG AA (4.5:1 text, 3:1 large text)
- [ ] Error messages programmatically associated with fields
- [ ] State changes announced to assistive technology
Component-Level Template
Component: [Name]
ARIA Pattern: [APG reference]
Keyboard:
- [ ] [Key]: [Expected behavior]
Screen Reader:
- [ ] Role announced as: [expected role]
- [ ] Name announced as: [expected name]
- [ ] State changes announced: [list states]
Visual:
- [ ] Focus indicator visible (3:1 contrast)
- [ ] Color contrast meets AA
- [ ] Works at 200% zoom
Definition of Done Checklist
- Automated scan passes (axe-core, zero violations)
- Keyboard-only operation verified (full workflow)
- Focus management correct (modals, route changes, dynamic content)
- Screen reader announces all content meaningfully (NVDA or VoiceOver)
- Color contrast meets AA thresholds
- Content reflows at 200% zoom
- Error states announced and associated with fields
- Heading hierarchy logical (no skipped levels)
Full templates: references/acceptance-criteria-templates.md
Sources: TetraLogical, Section508.gov, TPGi
7. Common Mistakes
See: references/common-mistakes.md
The most frequent testing anti-patterns:
"Axe passes, so we're accessible" — Automation catches at most 57% of issues by volume, and only 13% of WCAG criteria with high confidence. (Deque, Accessible.org)
Testing only with one screen reader — JAWS, NVDA, and VoiceOver behave differently. A component working in one may fail in another. Test with at least two. (WebAIM Survey #10)
Skipping keyboard testing — The fastest, cheapest tier. No tools needed. Catches focus traps, missing operability, and broken tab order that automation misses entirely.
Testing at the end — Accessibility debt compounds. Test each tier as part of the PR workflow, not as a pre-release gate.
Writing vague acceptance criteria — "Must be accessible" is not testable. Specify what the screen reader should announce, what keyboard interaction is expected, and what contrast ratio applies.
Ignoring mobile — 91.3% of screen reader users use one on mobile. VoiceOver (iOS) and TalkBack (Android) have different interaction models than desktop. (WebAIM Survey #10)
Not testing with real assistive technology — Browser DevTools accessibility inspectors show the accessibility tree but not what users actually hear. Run NVDA or VoiceOver and listen.
8. Cross-References
a11y-audit— Full audit methodology and WCAG conformance checkingaria-decision-framework— When and how to use ARIA correctlyfocus-management— Focus trapping, restoration, and roving tabindex patternsform-a11y— Form labeling, error handling, and validation patternslive-regions—aria-liveregion patterns and announcement timing