UI Tap

UI Tap

Overview

This skill allows Claude to programmatically trigger UI elements in the running iOS or Android app by sending HTTP requests to the test server. Each UI element is registered with a unique ID and can be triggered remotely, enabling automated UI interaction and testing workflows.

When to Use

Invoke this skill when you need to:

• Press a button or interact with a UI element programmatically
• Test UI interactions without manual intervention
• Verify that a UI change produces the expected result
• Coordinate button presses with screenshot capture
• Automate multi-step UI workflows

Trigger phrases:

• "tap the [element] button"
• "press [element]"
• "trigger [element]"
• "click [element]"
• "interact with [element]"

Prerequisites

1. Port forwarding must be active: The test server on port 8081 must be forwarded from the device to localhost:

   # iOS (keep running in background)
   pymobiledevice3 usbmux forward 8081 8081 &
   
   # Android
   adb forward tcp:8081 tcp:8081
   

2. App must be running: The iOS or Android app must be running on the connected device with the test server active.

3. Element must be registered: The UI element must have been registered with the UIAutomationRegistry using a unique ID. Common registered elements include:

   • toolbar-home - Home button in toolbar
   • toolbar-plus - New post button in toolbar
   • toolbar-search - Search button in toolbar
   • toolbar-profile - Profile button in toolbar

Instructions

1. Verify Prerequisites

Check that port forwarding is active and the app is running:

curl http://localhost:8081/test/ping

Should return "succeeded". If not, set up port forwarding first.

2. Identify Element ID

Determine the ID of the UI element you want to trigger. Element IDs are defined in the app code when registering with UIAutomationRegistry. Common patterns:

• Toolbar buttons: toolbar-[icon-name] (e.g., toolbar-plus, toolbar-home)
• Custom elements: Check the registration code in the relevant View file

3. Trigger the Element

Send a POST request to the test server:

curl -X POST 'http://localhost:8081/test/tap?id=ELEMENT_ID'

Replace ELEMENT_ID with the actual element identifier.

4. Verify Response

The response will be JSON indicating success or failure:

Success:

{"status": "success", "id": "toolbar-plus"}

Failure (element not found):

{"status": "error", "message": "Element not found: invalid-id"}

5. Optional: Capture Screenshot

After triggering the element, capture a screenshot to verify the UI change:

# iOS
/Users/asnaroo/Desktop/experiments/miso/miso/platforms/ios/development/screen-capture/imp/screenshot.sh /tmp/ui-result.png

# Android
adb exec-out screencap -p > /tmp/ui-result.png

Then read the screenshot to verify the expected UI change occurred.

Example Workflows

Trigger New Post Editor

# Tap the + button to open new post editor
curl -X POST 'http://localhost:8081/test/tap?id=toolbar-plus'

# Capture screenshot to verify editor appeared
/Users/asnaroo/Desktop/experiments/miso/miso/platforms/ios/development/screen-capture/imp/screenshot.sh /tmp/new-post-editor.png

Navigate Home

# Tap home button to return to recent posts view
curl -X POST 'http://localhost:8081/test/tap?id=toolbar-home'

# Verify we're on the home view
/Users/asnaroo/Desktop/experiments/miso/miso/platforms/ios/development/screen-capture/imp/screenshot.sh /tmp/home-view.png

Multi-Step Workflow

# 1. Navigate to profile
curl -X POST 'http://localhost:8081/test/tap?id=toolbar-profile'
sleep 0.5  # Wait for navigation

# 2. Open new post editor from profile
curl -X POST 'http://localhost:8081/test/tap?id=toolbar-plus'
sleep 0.5  # Wait for sheet to appear

# 3. Verify final state
/Users/asnaroo/Desktop/experiments/miso/miso/platforms/ios/development/screen-capture/imp/screenshot.sh /tmp/profile-new-post.png

Expected Behavior

1. Immediate execution: The UI action should occur within milliseconds of the HTTP request
2. Main thread safety: All UI actions are automatically dispatched to the main thread
3. No app restart needed: The automation system is always active once the app is running
4. Visual feedback: Most UI actions produce visible changes (button highlights, sheets appearing, navigation)
5. Idempotent: Multiple taps of the same element should be safe (though effects may differ)

Troubleshooting

"Connection refused" or curl fails

Problem: Port forwarding is not active or test server is not running.

Solutions:

1. For iOS: Run pymobiledevice3 usbmux forward 8081 8081
2. For Android: Run adb forward tcp:8081 tcp:8081
3. Verify app is running on device
4. Test basic connectivity: curl http://localhost:8081/test/ping

"Element not found" error

Problem: The element ID is not registered or misspelled.

Solutions:

1. Check the element registration code in the View file (e.g., Toolbar.swift)
2. Verify the exact ID string (case-sensitive)
3. Ensure the view has appeared (registration often happens in .onAppear)
4. Check app logs for registration messages: [TESTSERVER] prefix

Action triggers but wrong behavior

Problem: The registered action doesn't match expectations.

Solutions:

1. Review the action closure in the registration code
2. Check if state bindings are correct
3. Verify the action is using the correct callbacks
4. Add logging inside the action closure for debugging

Screenshot doesn't show expected change

Problem: Screenshot captured before UI update completed.

Solutions:

1. Add a small delay before screenshot: sleep 0.5
2. For sheets/modals, use longer delay: sleep 1.0
3. For animations, wait for animation duration
4. Capture multiple screenshots to see transition

Technical Details

How It Works

1. Registration: UI elements register actions with UIAutomationRegistry.shared.register(id:action:)
2. Storage: Actions stored in thread-safe dictionary with concurrent queue
3. HTTP Endpoint: TestServer handles POST /test/tap?id=X requests
4. Lookup: TestServer queries registry for the element ID
5. Execution: Action dispatched to main thread via DispatchQueue.main.async
6. Response: JSON response indicates success or failure

Platform Support

• iOS: Fully implemented in UIAutomationRegistry.swift and TestServer.swift
• Android: Fully implemented in UIAutomationRegistry.kt and TestServer.kt

Element Registration Pattern

Recommended: View Modifier Pattern (Clean, declarative)

In SwiftUI views, use the .uiAutomationId() modifier directly on buttons or other interactive elements:

Button(action: {
    // Normal button action
    isEditing = true
}) {
    Image(systemName: "pencil.circle.fill")
}
.uiAutomationId("edit-button") {
    // Automation action (usually same as button action)
    isEditing = true
}

Benefits:

• No state management in ViewModels required
• Annotation lives right next to the UI element definition
• Automatically registers on .onAppear, no manual registration needed
• Works with SwiftUI's struct-based view system

Legacy: Manual Registration Pattern (Verbose, requires plumbing)

In SwiftUI views:

.onAppear {
    UIAutomationRegistry.shared.register(id: "unique-id") {
        // Action to perform (state changes, navigation, etc.)
    }
}

Note: Manual registration is still useful for non-button elements like gesture recognizers or complex views. For buttons, prefer the modifier pattern.

In Kotlin composables (recommended pattern):

@Composable
fun MyButton() {
    // Register for automation (auto-unregisters on dispose)
    RegisterUIElement("my-button") {
        // Action to perform
    }

    Button(onClick = { /* same action */ }) {
        Text("Click")
    }
}

Or using the modifier extension:

Button(
    /* action */ },
    modifier = Modifier.uiAutomationId("my-button") { /* action */ }
) {
    Text("Click")
}

Notes

• Element IDs should be descriptive and prefixed by component (e.g., toolbar-plus, profile-edit)
• Actions should be idempotent where possible
• Complex workflows may need delays between steps for animations
• Screenshots are the best way to verify UI changes
• This system is for testing/automation only, not production features
• All registered elements are logged when the test server starts

Related Skills

• ios-deploy-usb - Deploy app with new UI elements
• iphone-screen-capture - Continuous screen mirroring
• update-skill - Improve this skill based on usage

Additional Endpoints

List Registered Elements

curl http://localhost:8081/test/list-elements

Response:

{"elements": ["toolbar-plus", "refresh-button"], "textFields": ["search-field"]}

Set Text in a Field

curl -X POST 'http://localhost:8081/test/set-text?id=search-field&text=hello'

Response:

{"status": "success", "id": "search-field", "text": "hello"}

Future Enhancements

• Support for element state queries (is button enabled? is view visible?)
• Batch operations (trigger multiple elements in sequence)
• Screenshot comparison (verify expected vs actual)
• Record and replay interaction sequences