Android E2e Verification

Android E2E Verification (Maestro)

Overview

"It compiles and the unit tests pass" is not proof that a feature works. This skill closes the loop: every feature slice gets a Maestro flow — a YAML file of user actions and assertions derived from the spec's acceptance criteria — that runs black-box against the installed app. Implement, build, run the flow, watch it pass (or fix and re-run). The flow is committed next to the code, so the acceptance criterion stays executable forever.

Maestro drives the app over adb with no app-code changes, no test hooks, and no build instrumentation — it works on release builds and survives refactors that would break selector-heavy tests.

When to Use

• Completing any feature slice with a user-visible flow (see incremental-implementation)
• Turning a spec's acceptance criteria into executable checks (see spec-driven-development)
• Verifying a bug fix actually fixes the user-facing behavior, not just the unit under test
• Regression-protecting critical journeys (login, checkout, sync) in CI
• Smoke-testing release candidates before rollout (see shipping-and-launch)

Skip when: The change has no runtime UI surface (pure data-layer refactor, build config) — unit/integration tests are the right layer. Don't use Maestro to test in-app logic permutations; that's what ViewModel and repository tests are for (see test-driven-development).

Core Process

Step 1: Install and Probe

1. Install Maestro (single binary, Java 17+):

# Pin the version so CI and local runs agree
export MAESTRO_VERSION=2.6.1
curl -fsSL "https://get.maestro.mobile.dev" | bash
# or: brew install mobile-dev-inc/tap/maestro

maestro --version   # probe availability

If Maestro is unavailable in the environment, say so explicitly and fall back to Compose/Espresso tests plus android screen capture verification — never silently skip E2E verification.

Step 2: Write the Flow From Acceptance Criteria — Before Implementing

2. Translate each acceptance criterion into a flow under .maestro/, named after the slice:

# .maestro/create-task.yaml
# Acceptance: "User can create a task and sees it in the list"
appId: com.example.tasks
---
- launchApp:
    clearState: true
- tapOn: "Add task"
- inputText: "Buy groceries"
- tapOn: "Save"
- assertVisible: "Buy groceries"

3. Flow rules:

   • One flow per acceptance criterion; compose shared steps with runFlow:

     - runFlow: subflows/login.yaml
     

   • clearState: true on launchApp for deterministic starts

   • Assert on user-visible text or content descriptions — the same things a user sees

   • Wrap genuinely async steps in retry blocks instead of sprinkling waits:

     - retry:
         maxRetries: 3
         commands:
           - tapOn: "Sync"
           - assertVisible: "Synced"
     

   • Deterministic selectors first. assertWithAI: "the task list shows one completed item" is allowed only where a selector cannot express the assertion (visual patterns, dynamic third-party content) — AI assertions cost more and can flake

Step 3: Run the Loop

4. Implement → build/install → run → fix → re-run:

./gradlew installDebug          # or: android run --apks=...
maestro test .maestro/create-task.yaml

A failing flow is the signal to keep working; a passing flow is the tangible evidence the slice is done. Capture evidence for the PR when useful:

maestro record .maestro/create-task.yaml   # video of the run

5. Agent-driven loop (MCP): when the harness supports MCP, run maestro mcp to expose the device to the agent directly — the agent can tap, assert, and inspect live (Maestro Viewer) instead of shelling out per command. In headless/CI contexts, plain maestro test is the fallback. For ad-hoc exploration without Maestro, use android screen capture --annotate + android screen resolve (see references/android-cli-reference.md).

Step 4: Wire Into CI

6. Run flows in the emulator job (see ci-cd-and-automation):

- name: E2E flows
  uses: reactivecircus/android-emulator-runner@v2
  with:
    api-level: 36
    arch: x86_64
    script: |
      ./gradlew installDebug
      maestro test .maestro/

maestro test .maestro/ runs every committed flow — the acceptance criteria of all shipped slices become the regression suite. For device-farm scale, maestro cloud runs the same flows on hosted devices.

Step 5: Choose the Right E2E Layer

7. Maestro is one layer, not the only one:

Deterministic YAML sits between fully-scripted in-process tests and fully-AI Journeys: resilient like AI, repeatable like code.

Common Rationalizations

Red Flags

• A completed feature slice with no flow under .maestro/
• Flows that only launchApp and assert nothing
• assertWithAI used where assertVisible would do
• Sleep-style waits instead of retry blocks
• Flows passing locally but not wired into the CI emulator job
• "Verified manually" in a PR description with no recorded or committed flow
• Maestro testing logic permutations that belong in ViewModel unit tests

Verification

• Every acceptance criterion of the slice has a flow in .maestro/
• maestro test .maestro/<slice>.yaml passes against the current build (paste the output)
• Flows start from clearState: true (or document why not)
• Assertions target user-visible text/content descriptions
• assertWithAI only where a selector cannot express the assertion
• CI runs maestro test .maestro/ in the emulator job
• If Maestro was unavailable, the fallback verification used is stated explicitly