Mobile Testing Strategy

Mobile Testing Strategy

Instructions

Mobile testing has a sharper pyramid than web or backend because simulators are imperfect, devices are expensive, and flakiness kills developer trust. A good strategy maximizes signal per minute of CI.

1. The Pyramid

              E2E on device
             /              \
         UI / widget tests
        /                    \
   Integration (repo + fake net)
  /                                 \
Unit tests (domain, reducers, utils)  <-- the broad base

Rough ratios for a healthy app: 70% unit, 20% integration, 7% UI, 3% E2E.

2. Unit Tests

• Target: pure logic (reducers, view models, domain rules, date math, parsers).
• No platform dependencies; no real clock; no real network.
• Run under 5 minutes for the whole suite.
• Frameworks: XCTest / Swift Testing, JUnit5 + MockK, package:test + mocktail, Jest / Vitest.

@Test func counter_increments() async throws {
    let vm = CounterVM(api: FakeApi(succeed: true))
    await vm.increment()
    #expect(vm.count == 1)
}

@Test fun counter_increments() = runTest {
    val vm = CounterVM(FakeApi(succeed = true))
    vm.increment()
    assertEquals(1, vm.state.value.count)
}

3. Integration Tests

• Exercise the repository with a fake network layer and a real (in-memory) database.
• Verify caching, offline, retry, and error mapping.
• Still no UI. Fast.
• This tier catches the most bugs per minute.

4. UI / Widget Tests

• Single-screen tests that render the UI with a fake view model.
• Assert on observable state and user-facing text, not internal widget types.
• Frameworks: SwiftUI snapshot + ViewInspector, Compose createComposeRule, flutter_test, React Native Testing Library.
• Use screenshot/golden tests for the design system only, not every screen.

5. End-to-End (on device)

• A small set of critical user journeys (sign-in, purchase, submit) running on real devices.
• Frameworks: XCUITest, Espresso / UI Automator, integration_test (Flutter), Maestro, Detox (RN), Appium (polyglot).
• Target 5-15 flows total. More is unsustainable.

6. Device Selection

A minimal matrix that catches most issues:

Rotate quarterly. Device clouds (BrowserStack, Firebase Test Lab, AWS Device Farm, Sauce Labs, HeadSpin) handle the long tail.

7. Flake Reduction

Flaky tests are worse than no tests. Treat flake rate above 1% as a bug.

• Eliminate real time. Inject a clock, use TestDispatcher, fakeAsync, virtual schedulers.
• Eliminate real network. Fakes and recorded fixtures, not live servers.
• Use idling resources (Espresso) and waitUntil on visible text, not sleeps.
• Isolate state. Reset DB, keychain, and preferences between tests.
• Quarantine, do not ignore. Tag flaky tests and triage weekly; delete or fix.

8. Coverage as a Signal

Coverage numbers are useful as a trend, not a gate. Require it to not decrease on changed files. Ignore coverage of generated code and platform glue. Chase critical-path coverage, not total.

9. CI Structure

• On every push: unit + integration + static analysis. Under 10 minutes.
• On PR: add UI tests on one emulator per platform.
• On main merge: full E2E matrix on device cloud; golden tests; size and startup budgets.
• Nightly: extended E2E, memory leak tests, stress tests.

Keep PR CI fast. Developers will route around slow CI, and the suite will rot.

10. Anti-Patterns

• Huge E2E suites. They go red on network flakes, not product regressions.
• Testing platform classes (e.g., UILabel rendering). Trust the platform.
• One giant "integration test" that covers five screens. Split.
• Snapshot tests of every screen. Restrict to the design system.
• Pinning to a single device. Test the matrix.
• Running UI tests against a live staging backend. Use contract tests and fakes.

11. Observability in Tests

• Capture screenshots on failure.
• Capture log output of the device + app.
• On E2E failures, attach a video when the framework supports it.
• Auto-open issues for new failures with rich context; do not ping the team for flakes.

Checklist

• Pyramid ratios roughly match 70/20/7/3.
• Unit tests run in under 5 minutes locally.
• Integration layer uses fakes for network and in-memory DB.
• UI/widget tests assert on observable state, not platform internals.
• E2E covers 5-15 critical journeys.
• Device matrix includes latest, mid, oldest, tablet, and low-memory.
• Flake rate is tracked; anything above 1% is triaged.
• CI separates PR-fast from main-thorough lanes.
• Failures capture screenshots, logs, and videos where possible.