Android UI Verify

Animeko Android UI Verification

Drive the real app on a real Android runtime. The core loop is: act → screenshot → Read the PNG → decide next action. Never claim a UI behavior works without a screenshot that shows it.

All commands below run from the ani repo root. The toolbox script is:

DROID=.agents/skills/android-ui-verify/scripts/droid.sh

1. Setup: emulator + app

$DROID emulator     # starts (or reuses) Pixel_8_Pro_API_35 headless, waits for boot, disables animations

• Must be an API 35+ AVD: this checkout's debug APK has minSdkVersion 35. Default AVD is Pixel_8_Pro_API_35; override with ANI_AVD=.... HEADED=1 $DROID emulator shows the window if the user wants to watch.
• Boot takes ~30–90 s. Emulator log: /tmp/animeko-emulator-<AVD>.log.

Build and install (one Gradle task does both):

./gradlew :app:android:installDefaultDebug -Pani.android.abis=arm64-v8a

• Expect Installed on 1 device. A full build can take several minutes; if an up-to-date APK already exists under app/android/build/outputs/apk/default/debug/, $DROID install adb-installs it directly.
• Installed debug package: me.him188.ani.debug (launcher activity me.him188.ani.android.activity.MainActivity).
• If validation needs a real API server, start the local one (see ../ani-api-server skill animeko-server-local-test-server) and build with -Pani.api.server=http://10.0.2.2:4394 — 10.0.2.2 is the emulator's alias for the host's localhost.

2. The interaction loop

$DROID launch                 # or: relaunch (force-stop + start), clear (wipe data → first-run onboarding)
sleep 8                       # first launch is slow; later navigations need 1–3 s
$DROID screenshot             # prints a PNG path — ALWAYS Read that file to actually see the screen

To interact, get coordinates from the UI hierarchy, not by guessing pixels off the screenshot:

$DROID find 追番              # searches text/content-desc/resource-id, prints tap-ready center=(x,y)px + dp size
$DROID tree                   # indented Compose semantics tree with dp sizes/positions (--all includes plain containers)
$DROID tap 540 1200
$DROID swipe 540 2000 540 800 300      # scroll up (drag content upward)
$DROID text hello                      # ASCII only — `adb input text` cannot type CJK; avoid tests requiring Chinese IME input
$DROID back                            # keyevent 4; also: key <code>, home
$DROID ui                              # raw hierarchy XML path, grep it when tree/find aren't enough

tree and find read the app-level Compose semantics tree (Compose projects it onto the accessibility tree, which uiautomator dumps). What it can and cannot tell you:

• Visible: all semantic text, contentDescription, exact on-screen bounds (printed both as dp — WxHdp@(x,y)dp — and px center), clickable/scrollable/selected/checked/disabled state.
• Not visible: composable function names, modifier chains, padding/arrangement parameters, colors. Colors and visual style are verified from screenshots; for composable-level internals use Android Studio's Layout Inspector manually.
• The repo's Modifier.testTags DO show up as resource-id in debug builds (e.g. find buttonNextStep → id=buttonNextStep in the onboarding wizard) — MainActivity enables testTagsAsResourceId when BuildConfig.DEBUG. Prefer testTag ids over visible text when they exist; note they only cover ~44 spots (wizard nav, video player controls).
• Many container nodes are unlabeled android.view.View; tree hides the purely structural ones by default.
• After every action that should change the screen: sleep 1-3 (longer for network content), then screenshot + Read, and compare against what you expected. If the screen didn't change, re-check coordinates with find before retrying.
• App misbehaving? $DROID logcat 150 (app log tail) and $DROID crashes (crash buffer) before drawing conclusions.

3. Responsive / wide-screen checks

$DROID display wide      # 2560x1600 @ 320dpi ≈ PC-like 1280x800dp; also: tablet, phone, reset
$DROID relaunch && sleep 5
$DROID screenshot

Report the active wm size/wm density with such screenshots. This validates responsive Compose layout on Android; it does not validate desktop-only code paths (for those, see .agents/skills/desktop-ui-verify).

4. Figma design comparison

When asked whether a screen matches its Figma design:

1. Get the design reference. If a Figma MCP server is available, call get_screenshot on the frame/node URL for the visual, and get_design_context (or get_variable_defs) for exact colors, spacing, radii, and typography. Otherwise ask the user for an exported PNG of the frame.
2. Match the runtime viewport to the frame before capturing: pick the closest display preset ($DROID display phone|tablet|wide), relaunch, and navigate to the target screen.
3. $DROID screenshot, then Read both images and compare. Do NOT expect a pixel-perfect diff — status bar, fonts, and rendering always differ. Compare structurally:
   • Layout & sizes: $DROID tree prints the runtime Compose semantics tree with dp sizes/positions — Figma values are already in dp, so element sizes, spacing, and ordering can be checked numerically against the design, no unit conversion needed.
   • Content: exact texts. $DROID find <text> proves presence and gives that element's dp size + px center.
   • Style: colors, corner radii, icon shapes — judge against get_design_context values plus the screenshots (these are NOT in the semantics tree).
4. Report a per-item verdict list (match / mismatch + what differs), citing both image paths.

5. Logs

$DROID logcat 150     # tail of the app process's logcat (falls back to crash buffer if not running)
$DROID crashes        # crash buffer only (`logcat -b crash`)

Use these whenever behavior looks wrong before drawing conclusions, and attach relevant lines to findings.

6. Screen recording — catching transient glitches

Some bugs are visible for well under a second: a seek bar that jumps to a wrong position and back, a flash of empty/mis-styled content, a one-frame layout shift during a transition. Screenshots will miss them — record the screen and diff the frames instead:

REC=$($DROID record-start)     # starts recording in background, prints the output .mp4 path
$DROID tap ... / swipe ...     # do the interaction that should (or should not) glitch
$DROID record-stop             # stops, pulls, prints the video path
$DROID frames "$REC"           # scans all frames, prints change events + exported images

frames (= scripts/frame_diff.py, needs ffmpeg on PATH) scores every consecutive-frame difference, groups spikes into events, and per event exports a contact sheet (frames tiled left→right, top→bottom — Read it to see the whole sequence at a glance) plus full-res frames named by video timestamp. How to work with it:

• Recordings are VFR: frames are written only when the screen changes, so a gap in timestamps proves the screen was static there. Any frame = something changed.
• Judge events against your own actions. Count and order your taps: an event when nothing should have changed, or a change-then-revert pair in quick succession, is the glitch. Timestamps are video-relative (recording starts ~1–2 s before record-start returns), so correlate by order/spacing, not wall clock.
• Zoom into a moment with $DROID frames <video> --around <t> --window 0.5; print the full per-frame score timeline with --list.
• Small elements barely move the frame-wide score (a progress bar is ~2 % of pixels). Crop to the region of interest: --crop W:H:X:Y in video pixels, and/or lower --threshold (default 0.01, try 0.003).
• Recordings are made at half display resolution (--size display/2, 16-aligned — the emulator encoder rejects full size, and its unsized fallback produces a broken 1-frame video). So video pixel ≈ screenshot/tap pixel ÷ 2.
• screenrecord bakes the display's rotation in at start: if the app rotates mid-recording (e.g. entering the fullscreen player), the video letterboxes. Record within one orientation; record <seconds> (blocking) suits no-interaction captures like a launch animation.

7. Verify & report

• For each checked behavior state: the action taken, the expected result, and the screenshot path proving it.
• Screenshots land in ${TMPDIR}/animeko-ui-verify/ by default; pass an explicit path to keep them (e.g. under ../reaction-screenshots/<task>/).
• A blank/white screenshot right after launch usually means the app is still loading — wait and retake before concluding breakage.
• Verification fails ≠ tooling fails: report app crashes (with crashes output) as findings, not as skill errors.

8. Cleanup

Unless the user asked to keep the session running:

$DROID display reset
$DROID kill-emulator

Instrumented tests on the emulator

Application module:

./gradlew :app:android:connectedDefaultDebugAndroidTest

Kotlin Multiplatform Android library modules:

./gradlew :app:shared:app-data:compileAndroidDeviceTest
./gradlew :app:shared:app-data:connectedAndroidDeviceTest

Narrow to one class with runner args (note: the full device-test APK is still packaged, so dex/package failures elsewhere can still fail the task):

./gradlew :app:shared:app-data:connectedAndroidDeviceTest \
  -Pandroid.testInstrumentationRunnerArguments.class=<fully.qualified.TestClass>

Known blocker (still present as of 2026-07-05): :app:shared:app-data:connectedAndroidDeviceTest fails at dexing because some commonTest backtick method names are invalid dex names, e.g. OfflineDownloadMediaResolverTest's `resolve - without fallback, engine failure surfaces as MediaResolutionException`. Treat as a project test-compatibility blocker, not an emulator failure.

Environment facts

• Android SDK: $ANDROID_HOME, falling back to ~/Library/Android/sdk. adb is on PATH; emulator is not (script handles it).
• Existing AVDs on this machine: Pixel_2, Pixel_4a_API_30 (too old, minSdk 35 blocks install), Pixel_8_Pro_API_35.
• Multiple devices connected → set ANDROID_SERIAL=emulator-5554 (or pass -s to raw adb calls).
• $DROID help lists every subcommand.