Compose Driver

Compose Driver Skill

Description

The Compose Driver Skill enables AI agents to see and control any Jetpack Compose UI (Android or Desktop) via a standardized HTTP API. It wraps a target Composable in a test harness and exposes a local server that translates HTTP requests into ComposeUiTest actions.

Running the Driver

The driver must be running for the agent to interact with the UI. Before starting the driver, always ensure previous instances are killed to avoid port conflicts (the server uses port 8080).

• Server Address: http://localhost:8080
• Target: compose.driver.composable must be the fully qualified name of the Composable function (e.g., package.FileKt.ComposableName).
• Wait for Ready: After starting the command, poll GET /status until it returns "ok".

Desktop (Recommended for speed for JVM & Multiplatform apps):

pkill -f "compose-driver-desktop" || true; ./gradlew :compose-driver-desktop:run -Dcompose.driver.composable=io.github.jdemeulenaere.compose.driver.sample.desktop.DesktopApplicationKt.DesktopApplication

Android (Robolectric):

[!NOTE] The Android server will run in a Robolectric test environment, so start-up is expected to be slower (~5s on modern hardware).

pkill -f "compose-driver-android" || true; ./gradlew :compose-driver-android:run -Dcompose.driver.composable=io.github.jdemeulenaere.compose.driver.sample.android.AndroidApplicationKt.AndroidApplication

API Reference (Agent Actions)

The agent interacts with the UI by sending HTTP GET requests.

Common Parameters:

These parameters can be used on all endpoints, applying the operation to the matching node.

• nodeTag (Optional): The Modifier.testTag of the target node.
• nodeText (Optional): Matches a node by its text content.
• nodeTextSubstring (Optional): If true, nodeText matches as a substring (default: false, exact match).
• nodeTextIgnoreCase (Optional): If true, nodeText matching is case-insensitive (default: false).
• If both nodeTag and nodeText are provided, the node must match both. If neither is provided, the action applies to the root node.
• gifDurationMs (Optional): If provided (max 5000ms), the server records the interaction and returns a GIF instead of a text response. Requires ffmpeg.

1. Observation (Seeing the UI)

2. Synchronization (Waiting)

3. Interaction (Control)

Key Events

The /keyEvent endpoint sends raw keyboard events to a Compose UI node.

Parameters:

• key (required): Property name from androidx.compose.ui.input.key.Key (e.g., A, Enter, DirectionUp, ShiftLeft, F1).
• action (optional, default: press):
  • press — sends key down + key up (a full key press).
  • down — sends only key down (hold the key).
  • up — sends only key up (release the key).
• modifiers (optional): Comma-separated list of modifier key names to hold down during a press action (e.g., ShiftLeft, CtrlLeft,ShiftLeft). Only applies when action=press.

Examples:

GET /keyEvent?key=Enter
GET /keyEvent?key=A&nodeTag=myTextField
GET /keyEvent?key=A&modifiers=ShiftLeft
GET /keyEvent?key=C&modifiers=CtrlLeft
GET /keyEvent?key=Delete&modifiers=CtrlLeft,ShiftLeft
GET /keyEvent?key=ShiftLeft&action=down
GET /keyEvent?key=ShiftLeft&action=up

4. Gestures

5. Lifecycle

Usage Examples for Agents

Scenario: Login Flow

1. Inspect: GET /printTree -> Identify tags username_field, password_field, login_button.
2. Wait: GET /waitForNode?nodeTag=username_field
3. Act:
   • GET /textInput?nodeTag=username_field&text=myuser
   • GET /textInput?nodeTag=password_field&text=mypass
   • GET /click?nodeText=login&nodeTextIgnoreCase=true&nodeTextSubstring=true
4. Verify: GET /waitForNode?nodeTag=home_screen_title

Scenario: Debugging an Animation

1. Record: GET /click?nodeTag=animate_btn&gifDurationMs=1000
   • Result: Returns a GIF file showing the click and the subsequent 1 second of animation.

Scenario: Interacting with Off-Screen List Items

If printTree shows an element exists but is not visible because it is in a scrollable container and outside of the current viewport, you must scroll to it before interacting with it (e.g. clicking).

1. Inspect: GET /printTree -> Find the tag target_item.
2. Scroll: GET /scrollTo?nodeTag=target_item -> This brings the item into the viewport.
3. Act: GET /click?nodeTag=target_item

Scenario: Keyboard Shortcuts

1. Focus: GET /click?nodeTag=editor -> Click to focus the text editor.
2. Select all: GET /keyEvent?key=A&modifiers=CtrlLeft&nodeTag=editor
3. Copy: GET /keyEvent?key=C&modifiers=CtrlLeft&nodeTag=editor
4. Navigate: GET /keyEvent?key=DirectionDown&nodeTag=editor -> Move cursor down.
5. Submit: GET /keyEvent?key=Enter&nodeTag=editor

Source: jdemeulenaere/compose-driver — distributed by TomeVault.