# Gui Agent Interaction

> Implements GUI agent interaction patterns (screen vision recognition, UI element detection, automated mouse/keyboard execution) for operating desktop and web applications without APIs.

- Skill: `paulpas/gui-agent-interaction` (Agent Skill)
- Install (CLI): `npx skillmds@latest add paulpas/gui-agent-interaction`
- Raw SKILL.md: https://api.skillmd.com/api/skills/paulpas/gui-agent-interaction/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: MIT
- Author: paulpas (https://skillmd.com/u/paulpas)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/paulpas/gui-agent-interaction

---


# GUI Agent Interaction Pattern

Implements screen-based interaction pipelines so AI agents can operate desktop and web applications by "seeing" rendered UI elements through computer vision and executing mouse/keyboard actions — no native APIs required. This skill applies the 5 Laws of Elegant Defense: Law 1 (Early Exit) for guard-clause-driven action validation, Law 2 (Parse at boundary) for screen state normalization, and Law 3 (Atomic Predictability) for immutable before/after state snapshots used in verification loops.

This skill covers how to build agents that navigate graphical user interfaces end-to-end: capturing screenshots, detecting interactive elements via vision models, planning action sequences, executing them through OS-level input libraries, and verifying outcomes by comparing screen states before and after each step.

## TL;DR Checklist

- [ ] Choose the right interaction layer — browser automation (Playwright/Selenium) for web, PyAutoGUI for desktop, or a hybrid pipeline
- [ ] Implement screen capture with consistent resolution and color space (RGB, not RGBA) across all steps
- [ ] Run UI element detection on every captured frame before planning any action
- [ ] Execute actions through a typed execution engine that maps high-level intents to OS commands
- [ ] Verify every action by capturing a post-action screenshot and diffing against expected state changes
- [ ] Implement error recovery with timeout thresholds and fallback dialog classification
- [ ] Log full interaction traces (screenshots, detected elements, actions taken) for replay debugging

---

## When to Use

Use this skill when:

- Automating legacy applications with no REST API or programmatic interface (e.g., internal enterprise web portals built with server-side rendering)
- Interacting with desktop software where only GUI exposure exists (e.g., configuring a system administration tool on Windows/macOS/Linux)
- Performing form-filling workflows across multiple disconnected web applications that lack integration points
- Testing end-user experience of web or desktop applications by simulating real user interactions at the pixel level
- Validating visual correctness of UI changes — comparing rendered screens before and after a deployment or style update
- Building agents that must operate in environments where only screen-level access is permitted (air-gapped systems, restricted containers)

## When NOT to Use

Avoid this skill for:

- Applications with well-documented REST/GraphQL APIs — always prefer programmatic API calls over visual interaction (use `tool-use-function-calling` instead)
- High-frequency trading or latency-sensitive automation where screen capture overhead introduces unacceptable delay (milliseconds matter — use exchange adapters directly)
- Environments requiring pixel-perfect precision below 5-pixel tolerance (computer vision detection accuracy degrades with resolution; use DOM-based selectors when available)
- Accessibility-compliance testing that requires semantic markup validation — screen-level interaction cannot verify ARIA attributes or screen reader output (use accessibility-in-ui-adjacent-code)

---

## Core Workflow

```
┌─────────────┐    ┌──────────────────┐    ┌──────────────┐    ┌───────────────┐
│ Screen      │───→│ UI Element       │───→│ Action       │───→│ State         │
│ Capture     │    │ Recognition      │    │ Planning     │    │ Verification  │
│ (screenshot)│    │ (vision model →  │    │ (LLM maps    │    │ (before/after │
│             │    │  bounding boxes, │    │  intent →    │    │  diff check)  │
│             │    │  element types)  │    │  OS commands)│    │               │
└─────────────┘    └──────────────────┘    └──────────────┘    └───────┬───────┘
                                                                       │
                                                          ┌────────────▼────────┐
                                                          │ Error Recovery &   │
                                                          │ Retry Loop          │
                                                          └─────────────────────┘
```

1. **Capture Screen State** — Acquire a screenshot of the current visible UI surface at consistent resolution and color format:
    - Use browser automation APIs for web pages (Playwright's `screenshot()` or Selenium's `get_screenshot_as_file()`)
    - Use OS-level capture for desktop apps (mss for cross-platform, Quartz for macOS, GDI/DirectX for Windows)
    - Normalize to RGB format at a fixed resolution (1920x1080 minimum; scale smaller screens up consistently)
    **Checkpoint:** Every captured frame must be saved with a monotonic timestamp and stored alongside its element detection result.

2. **Detect UI Elements** — Run computer vision inference on the screenshot to identify all interactive elements with bounding boxes and classification labels:
    - Use a fine-tuned object detection model (YOLOv8, RT-DETR) trained on UI element taxonomies (buttons, inputs, links, menus, dialogs)
    - Alternatively use DOM scraping for web pages as a complementary ground-truth layer when JavaScript is available
    - Output structured element list: `[{"type": "button", "label": "Submit", "bbox": [x1, y1, x2, y2], "confidence": 0.94}]`
    **Checkpoint:** Element detection must return at least one actionable element per screen — empty detection triggers a re-capture with zoom adjustment.

3. **Plan Actions from Detected State** — Given the task goal and current element map, generate a sequence of atomic UI actions:
    - Feed the screenshot + element list + task description to an LLM that outputs structured action sequences
    - Each action must be typed (`click`, `type`, `scroll`, `drag`, `hover`, `right_click`) with concrete coordinates and optional text payload
    - Validate action feasibility before execution — e.g., cannot type into a non-editable element
    **Checkpoint:** Action sequence must be executable top-to-bottom without requiring human judgment mid-sequence.

4. **Execute Actions** — Map high-level actions to OS or browser commands through an execution engine:
    - Web: Playwright/Selenium locator-based actions (`.click()`, `.fill()`, `.select_option()`) or coordinate-based fallback
    - Desktop: PyAutoGUI functions (`pyautogui.click(x, y)`, `pyautogui.typewrite(text)`), optionally wrapped with safe guards
    - Include deliberate delays between actions (100–500ms default) to account for rendering and animation timing
    **Checkpoint:** Every executed action must log its type, target coordinates/selector, and execution duration.

5. **Verify State After Execution** — Capture a post-action screenshot and compare it against the expected outcome:
    - Use structural similarity (SSIM) or perceptual hash (pHash) to detect meaningful changes vs noise
    - Re-run element detection on the new frame to confirm the expected elements appeared/disappeared/changed state
    - If verification fails, classify the error type and route to the recovery handler
    **Checkpoint:** State verification must complete within a timeout window (default: 5 seconds) — stale screens indicate hung processes.

6. **Handle Errors and Recover** — When an action produces an unexpected screen state, classify and attempt recovery:
    - Detect common failure patterns: loading spinners, permission dialogs, connection errors, CAPTCHAs
    - Apply recovery strategies in priority order: retry (same action), cancel dialog → retry, wait for timeout → retry
    - After max retries exhausted, log the full interaction trace and raise a structured error with screenshot attachment
    **Checkpoint:** Recovery must never blindly loop — every retry path must have an independent success criterion.

---

## Implementation Patterns

### Pattern 1: Google Project Mariner Architecture (Full GUI Agent Pipeline)

Google Project Mariner demonstrated that agents can navigate graphical interfaces by combining screen capture, element recognition, and action execution in a tight feedback loop. The core architecture chains three stages: vision-based UI understanding, LLM-driven action planning, and low-level command execution with verification.

```python
import logging
from dataclasses import dataclass, field
from enum import Enum
from typing import Any
from datetime import datetime, timezone
import time

logger = logging.getLogger("gui.agent")


class ActionType(Enum):
    """Atomic UI action types."""
    CLICK = "click"
    DOUBLE_CLICK = "double_click"
    TYPE = "type"
    SCROLL_UP = "scroll_up"
    SCROLL_DOWN = "scroll_down"
    DRAG = "drag"
    HOVER = "hover"
    RIGHT_CLICK = "right_click"
    KEY_PRESS = "key_press"


class ActionStatus(Enum):
    PENDING = "pending"
    EXECUTED = "executed"
    FAILED = "failed"
    RETRYING = "retrying"
    RECOVERED = "recovered"


@dataclass
class UIElement:
    """Detected interactive element on screen with bounding box."""
    element_id: str
    element_type: str            # "button", "input", "link", "menu", "dialog", "image"
    label: str                   # Visible text or accessible name
    bbox: tuple[int, int, int, int]  # (x1, y1, x2, y2) in pixel coords
    confidence: float            # Detection model confidence (0.0 – 1.0)

    @property
    def center(self) -> tuple[float, float]:
        """Return the geometric center of the bounding box."""
        cx = (self.bbox[0] + self.bbox[2]) / 2
        cy = (self.bbox[1] + self.bbox[3]) / 2
        return (cx, cy)

    @property
    def width(self) -> int:
        return self.bbox[2] - self.bbox[0]

    @property
    def height(self) -> int:
        return self.bbox[3] - self.bbox[1]

    def contains_point(self, x: float, y: float) -> bool:
        """Check if a coordinate falls within this element's bounding box."""
        return (self.bbox[0] <= x <= self.bbox[2] and
                self.bbox[1] <= y <= self.bbox[3])


@dataclass
class ScreenState:
    """Immutable snapshot of a UI screen at a point in time."""
    timestamp: str               # ISO 8601 with UTC timezone
    screenshot_path: str         # Path to saved PNG file
    width: int
    height: int
    elements: list[UIElement] = field(default_factory=list)
    metadata: dict[str, Any] = field(default_factory=dict)


@dataclass
class PlannedAction:
    """An action planned by the agent for a specific target element."""
    action_type: ActionType
    target_element_id: str | None  # Which element this targets
    coordinates: tuple[float, float]  # (x, y) screen coordinates
    text_payload: str = ""           # For TYPE actions
    delay_ms: int = 200              # Wait between actions in the sequence
    expected_state_change: str = ""  # Description of what should happen after execution


@dataclass
class InteractionTrace:
    """Complete record of one agent interaction step."""
    step_index: int
    before_state: ScreenState | None = None
    action: PlannedAction | None = None
    action_status: ActionStatus = ActionStatus.PENDING
    after_state: ScreenState | None = None
    error_message: str | None = None
    recovery_action: str | None = None
    duration_ms: float = 0.0

    @property
    def is_complete(self) -> bool:
        return self.after_state is not None


class GUIAgentPipeline:
    """Implements the Google Project Mariner pipeline for GUI agent interaction.

    Chains screen capture → element recognition → action planning → execution →
    state verification into a loop. Applies Law 2 (Parse at boundary) by
    normalizing all screen captures to a consistent format before passing to
    downstream stages, and Law 1 (Early Exit) by validating each stage's
    output before proceeding to the next.
    """

    def __init__(
        self,
        vision_model: Any = None,
        executor: Any = None,
        verifier: Any = None,
        max_retries: int = 3,
        action_delay_ms: int = 200,
        verification_timeout_s: float = 5.0,
    ) -> None:
        self.vision_model = vision_model
        self.executor = executor
        self.verifier = verifier
        self.max_retries = max_retries
        self.action_delay_ms = action_delay_ms
        self.verification_timeout_s = verification_timeout_s
        self.trace: list[InteractionTrace] = []

    def run(self, task_description: str) -> list[InteractionTrace]:
        """Execute a task on the target GUI by cycling through the interaction loop.

        Args:
            task_description: Natural language description of what the agent should accomplish.

        Returns:
            List of InteractionTraces recording each step's before/after state and outcome.
        """
        step_index = 0
        iteration = 0
        max_iterations = 50  # Prevent infinite loops on stuck UIs

        while iteration < max_iterations:
            iteration += 1
            trace = InteractionTrace(step_index=step_index)

            # Stage 1: Capture screen state
            trace.before_state = self._capture_screen()
            if not trace.before_state or not trace.before_state.elements:
                trace.error_message = "Screen capture returned no detectable elements"
                trace.action_status = ActionStatus.FAILED
                self.trace.append(trace)
                logger.error("Step %d: No elements detected on screen", step_index)
                break

            # Stage 2: Plan actions
            action_sequence = self._plan_actions(
                task_description, trace.before_state
            )
            if not action_sequence:
                trace.error_message = "Action planner returned empty sequence"
                trace.action_status = ActionStatus.FAILED
                self.trace.append(trace)
                break

            # Stage 3-5: Execute each action with verification
            executed_any = False
            for action in action_sequence:
                result = self._execute_with_verification(
                    action, trace.before_state, step_index
                )
                if result.action_status == ActionStatus.FAILED and result.recovery_action:
                    # Attempt recovery
                    retry_result = self._attempt_recovery(result)
                    if retry_result is not None:
                        result = retry_result

                if result.action_status in (ActionStatus.EXECUTED, ActionStatus.RECOVERED):
                    trace.before_state = result.after_state  # Feed back into loop
                    executed_any = True

                self.trace.append(result)
                step_index += 1

            # Check if task is complete
            if not action_sequence or executed_any:
                trace.action_status = ActionStatus.EXECUTED
                self.trace.append(trace)
                break

        return self.trace

    def _capture_screen(self) -> ScreenState | None:
        """Capture current screen state with element detection."""
        # Implementation depends on target environment (browser vs desktop)
        raise NotImplementedError("Subclass and implement for your target platform")

    def _plan_actions(
        self, task: str, state: ScreenState
    ) -> list[PlannedAction]:
        """Plan action sequence from task description and current element map."""
        raise NotImplementedError("Subclass with LLM-powered planner")

    def _execute_with_verification(
        self, action: PlannedAction, before: ScreenState, step_idx: int,
    ) -> InteractionTrace:
        """Execute an action and verify its effect."""
        trace = InteractionTrace(step_index=step_idx)
        trace.action = action
        trace.before_state = before

        start = time.time()
        try:
            trace.after_state = self._capture_screen()
            trace.action_status = ActionStatus.EXECUTED
        except Exception as e:
            trace.error_message = str(e)
            trace.action_status = ActionStatus.FAILED
        trace.duration_ms = (time.time() - start) * 1000

        return trace

    def _attempt_recovery(
        self, failed_trace: InteractionTrace,
    ) -> InteractionTrace | None:
        """Attempt to recover from a failed action."""
        raise NotImplementedError("Implement recovery strategies")
```

**BAD vs GOOD: Pipeline Design**

```python
# ❌ BAD — No early exit on empty screen state; loops forever on hung UI
class BrokenGUIAgent:
    def run(self, task):
        while True:  # Never terminates
            screenshot = capture_screen()
            elements = detect_elements(screenshot)
            actions = plan(actions_for(task, elements))
            execute(actions)

# ✅ GOOD — Explicit max_iterations, guard clauses at every stage boundary,
# immutable traces for replay debugging (Law 3: Atomic Predictability)
class RobustGUIAgent:
    def run(self, task):
        for _ in range(50):  # Hard cap prevents infinite loops
            state = self._capture_screen()
            if not state or not state.elements:
                break  # Early exit: nothing actionable to do
            actions = self._plan_actions(task, state)
            if not actions:
                break
            ...
```

### Pattern 2: UI Element Detection & Recognition System

UI element detection maps raw pixel data into structured element catalogs that the action planner can reason about. For web applications, DOM-based detection is preferred (direct access to element properties, text content, and accessibility labels). For desktop apps without DOM exposure, computer vision models detect elements purely from screen pixels.

```python
import base64
import io
from dataclasses import dataclass, field

try:
    from selenium.webdriver.common.by import By
    from selenium.webdriver.remote.webdriver import WebDriver
    SELENIUM_AVAILABLE = True
except ImportError:
    SELENIUM_AVAILABLE = False


@dataclass
class DOMElementInfo:
    """Structured info extracted from a web page's DOM tree."""
    element_id: str
    tag_name: str
    role: str | None            # ARIA role (button, textbox, link, etc.)
    aria_label: str | None      # Accessible label
    text_content: str           # Visible text between tags
    is_visible: bool
    is_interactive: bool        # Has click handler or is a form control
    rect: dict[str, int]        # {"left", "top", "width", "height"} in viewport coords

    @property
    def center(self) -> tuple[float, float]:
        left = self.rect["left"]
        top = self.rect["top"]
        return (left + self.rect["width"] / 2, top + self.rect["height"] / 2)


@dataclass
class VisionElementInfo:
    """Structured info from a computer vision model detecting elements in pixels."""
    element_id: str
    element_type: str           # button, input_field, link, menu_item, dialog, icon
    label: str                  # Inferred text label from OCR or visual features
    bbox: tuple[int, int, int, int]  # (x1, y1, x2, y2) absolute pixel coords
    confidence: float           # Model detection confidence
    ocr_text: list[dict] = field(default_factory=list)  # Raw OCR results near bbox


class WebElementDetector:
    """Extracts structured element info from a browser page's DOM tree.

    This is the preferred detection method for web applications since it
    provides ground-truth accessibility information that vision models
    cannot reliably infer from pixels alone.
    """

    INTERACTIVE_TAGS = {"a", "button", "input", "select", "textarea", "summary"}
    ATTRIBUTES_TO_EXTRACT = {
        "type", "name", "role", "aria-label", "aria-hidden",
        "disabled", "readonly", "placeholder", "value",
    }

    def __init__(self, driver: WebDriver) -> None:
        self.driver = driver

    def detect_all_interactive_elements(self) -> list[DOMElementInfo]:
        """Find all interactive elements on the current page.

        Uses JavaScript evaluation to extract element properties directly
        from the DOM, which is faster and more reliable than iterating
        through Selenium's find_element calls.
        """
        script = """
        (function() {
            const interactiveTags = %TAGS;
            const attrsToRead = %ATTRS;
            const results = [];

            // Get all elements, filter to interactive ones
            const allElements = document.querySelectorAll('*');
            for (const el of allElements) {
                if (!interactiveTags.has(el.tagName.toLowerCase())) continue;
                if (el.hidden || el.getAttribute('aria-hidden') === 'true') continue;

                // Check if element is actually visible in the viewport
                const rect = el.getBoundingClientRect();
                if (rect.width === 0 && rect.height === 0) continue;

                const info: Record<string, string> = {};
                for (const attr of attrsToRead) {
                    const val = el.getAttribute(attr);
                    if (val) info[attr] = val;
                }

                results.push({
                    tag: el.tagName.toLowerCase(),
                    type: el.type || null,
                    ...info,
                    rect: {
                        left: Math.round(rect.left),
                        top: Math.round(rect.top),
                        width: Math.round(rect.width),
                        height: Math.round(rect.height),
                    },
                    text: (el.textContent || '').trim().slice(0, 200),
                });
            }

            return results;
        })()
        """ % (
            repr(set(self.INTERACTIVE_TAGS)),
            repr(self.ATTRIBUTES_TO_EXTRACT),
        )

        raw_results = self.driver.execute_script(script) or []

        elements: list[DOMElementInfo] = []
        for i, raw in enumerate(raw_results):
            element_id = f"web-el-{i}"
            tag = raw.get("tag", "")
            role = raw.get("role") or self._infer_role(tag, raw)
            aria_label = raw.get("aria_label") or raw.get("aria-label")

            # Determine if element is interactive based on attributes
            disabled = raw.get("disabled") == "true"
            readonly = raw.get("readonly") == "true"
            is_interactive = not disabled and tag in self.INTERACTIVE_TAGS

            elements.append(DOMElementInfo(
                element_id=element_id,
                tag_name=tag,
                role=role,
                aria_label=aria_label,
                text_content=raw.get("text", ""),
                is_visible=True,  # Already filtered by visibility check in JS
                is_interactive=is_interactive,
                rect=raw.get("rect", {}),
            ))

        return elements

    def _infer_role(self, tag: str, attrs: dict) -> str | None:
        """Infer ARIA role from HTML tag when role attribute is missing."""
        role_map = {
            "a": "link",
            "button": "button",
            "input[type='submit']": "button",
            "input[type='text']": "textbox",
            "input[type='email']": "textbox",
            "input[type='password']": "textbox",
            "select": "listbox",
            "textarea": "textbox",
        }
        return role_map.get(tag, None)


class VisionBasedElementDetector:
    """Detects UI elements from raw screen pixels using a vision model.

    Used when DOM access is unavailable (desktop apps, iframes with CORS blocks).
    Combines object detection (bounding boxes) with OCR (text labels).
    """

    def __init__(self, model: Any = None, ocr_engine: Any = None) -> None:
        self.model = model  # Object detection model (YOLOv8/RT-DETR)
        self.ocr_engine = ocr_engine  # OCR engine (Tesseract, EasyOCR)

    def detect_from_screenshot(self, image_data: bytes | io.BytesIO) -> list[VisionElementInfo]:
        """Run element detection on a screenshot image.

        Args:
            image_data: Raw PNG or JPEG image bytes from screen capture.

        Returns:
            List of VisionElementInfo with bounding boxes and OCR'd labels.
        """
        # Pass to object detection model for bounding boxes
        detections = self.model.predict(image_data) if self.model else []

        elements: list[VisionElementInfo] = []
        for det in detections:
            element_id = f"vis-el-{det.class_id}-{det.confidence:.3f}"
            bbox = (
                int(det.x1), int(det.y1),
                int(det.x2), int(det.y2),
            )

            # Extract text label via OCR on the cropped region
            ocr_text = []
            if self.ocr_engine and bbox[2] > bbox[0]:
                cropped = image_data[bbox[1]:bbox[3], bbox[0]:bbox[2]]
                ocr_text = self.ocr_engine.recognize(cropped)

            elements.append(VisionElementInfo(
                element_id=element_id,
                element_type=self._map_class_to_type(det.class_id),
                label=self._infer_label(ocr_text),
                bbox=bbox,
                confidence=float(det.confidence),
                ocr_text=ocr_text,
            ))

        return elements

    def _map_class_to_type(self, class_id: int) -> str:
        """Map model output class ID to UI element type name."""
        type_map = {
            0: "button",
            1: "input_field",
            2: "link",
            3: "menu_item",
            4: "dialog",
            5: "image",
            6: "icon",
            7: "tab",
            8: "dropdown",
            9: "checkbox",
        }
        return type_map.get(class_id, "unknown")

    def _infer_label(self, ocr_results: list[dict]) -> str:
        """Extract the most prominent text label from OCR results."""
        if not ocr_results:
            return ""
        # Sort by confidence and return top result
        sorted_results = sorted(ocr_results, key=lambda x: x.get("confidence", 0), reverse=True)
        return sorted_results[0].get("text", "") if sorted_results else ""
```

**BAD vs GOOD: Element Detection**

```python
# ❌ BAD — Relies on fragile CSS selectors that break with UI updates
driver.find_element(By.CSS_SELECTOR, "#main-content > div:nth-child(3) > button")

# ✅ GOOD — Semantic element matching by accessible role and visible text,
# resilient to DOM restructuring as long as the label stays the same
driver.find_element(By.XPATH, "//button[normalize-space()='Submit Order']")

# ❌ BAD — No confidence threshold; processes low-confidence detections as real elements
detections = vision_model.predict(screenshot)  # All results treated equally
for det in detections:
    actions.append(PlannedAction(target=det))

# ✅ GOOD — Only high-confidence detections feed into action planning
HIGH_CONFIDENCE_THRESHOLD = 0.85
actions = [
    PlannedAction(target=det)
    for det in detections
    if det.confidence >= HIGH_CONFIDENCE_THRESHOLD
]
```

### Pattern 3: Action Execution Engine (Click, Type, Scroll, Drag)

The execution engine translates high-level action plans into OS or browser commands. For web automation, Playwright is preferred over Selenium due to its auto-wait capabilities and locator resolution system that reduces flakiness. For desktop automation, PyAutoGUI provides cross-platform input simulation with built-in safety guards.

```python
import logging
import time
from dataclasses import dataclass

try:
    from playwright.sync_api import Page, sync_playwright
    PLAYWRIGHT_AVAILABLE = True
except ImportError:
    PLAYWRIGHT_AVAILABLE = False

try:
    import pyautogui
    PYAUTOGUI_AVAILABLE = True
    # Safety guards — must be set on first run
    if not hasattr(pyautogui, '_PAUSES_SET'):
        pyautogui.PAUSE = 0.25          # Pause between actions (seconds)
        pyautogui.FAILSAFE = True       # Move mouse to corner to abort
        pyautogui.MINIMUM_DURATION = 0.1
        pyautogui._PAUSES_SET = True
except ImportError:
    PYAUTOGUI_AVAILABLE = False

logger = logging.getLogger("gui.execution")


@dataclass
class ExecutionResult:
    """Outcome of a single action execution attempt."""
    success: bool
    action_type: str
    target_description: str
    duration_ms: float
    error_message: str | None = None
    screenshot_path: str | None = None  # Post-action screenshot for debugging


class PlaywrightExecutionEngine:
    """Executes UI actions via Playwright's browser automation API.

    Preferred over Selenium for web automation due to auto-wait (automatically
    waits for elements to be actionable before clicking/filling), network event
    interception, and modern locator strategies (CSS, XPath, text, role-based).
    """

    def __init__(self, page: Page) -> None:
        self.page = page

    def click(self, selector: str, *, timeout_ms: int = 10_000) -> ExecutionResult:
        """Click an element identified by a Playwright locator.

        Auto-waits for the element to be visible and actionable before clicking.
        This eliminates the need for explicit sleep/delay calls.

        Args:
            selector: CSS selector, XPath, or text/role-based locator string.
            timeout_ms: Maximum time to wait for element readiness.

        Returns:
            ExecutionResult with success status and timing.
        """
        start = time.time()
        try:
            # Playwright's .click() auto-waits — no explicit delay needed
            self.page.locator(selector).click(timeout=timeout_ms)
            duration = (time.time() - start) * 1000
            return ExecutionResult(
                success=True,
                action_type="click",
                target_description=f"element matched: {selector}",
                duration_ms=round(duration, 2),
            )
        except Exception as e:
            logger.warning("Click failed on '%s': %s", selector, e)
            return ExecutionResult(
                success=False,
                action_type="click",
                target_description=f"element matched: {selector}",
                duration_ms=round((time.time() - start) * 1000, 2),
                error_message=str(e),
            )

    def type_into(
        self, selector: str, text: str, *, delay_per_char_ms: int = 50,
    ) -> ExecutionResult:
        """Type text into an input field with character-by-character delay.

        Args:
            selector: Locator for the target input element.
            text: Text to type.
            delay_per_char_ms: Artificial delay between characters (simulates human typing).

        Returns:
            ExecutionResult with success status and timing.
        """
        start = time.time()
        try:
            input_el = self.page.locator(selector)
            input_el.click(timeout=5000)          # Focus the field first
            input_el.fill(text)                    # Fill replaces existing content
            time.sleep(len(text) * delay_per_char_ms / 1000)  # Human-like typing feel
            return ExecutionResult(
                success=True,
                action_type="type",
                target_description=f"field matched: {selector}",
                duration_ms=round((time.time() - start) * 1000, 2),
            )
        except Exception as e:
            return ExecutionResult(
                success=False,
                action_type="type",
                target_description=f"field matched: {selector}",
                duration_ms=round((time.time() - start) * 1000, 2),
                error_message=str(e),
            )

    def select_option(self, selector: str, value: str) -> ExecutionResult:
        """Select an option in a <select> dropdown element.

        Args:
            selector: Locator for the <select> element.
            value: The option value to select.
        """
        start = time.time()
        try:
            self.page.locator(selector).select_option(value=value)
            return ExecutionResult(
                success=True,
                action_type="select_option",
                target_description=f"dropdown matched: {selector}",
                duration_ms=round((time.time() - start) * 1000, 2),
            )
        except Exception as e:
            return ExecutionResult(
                success=False,
                action_type="select_option",
                target_description=f"dropdown matched: {selector}",
                duration_ms=round((time.time() - start) * 1000, 2),
                error_message=str(e),
            )

    def scroll(self, direction: str = "down", amount: int = 300) -> ExecutionResult:
        """Scroll the viewport by a pixel amount.

        Args:
            direction: 'up' or 'down'.
            amount: Pixels to scroll.
        """
        start = time.time()
        try:
            delta = -amount if direction == "up" else amount
            self.page.evaluate(f"window.scrollBy(0, {delta})")
            time.sleep(0.3)  # Allow lazy-loaded content to render
            return ExecutionResult(
                success=True,
                action_type="scroll",
                target_description=f"{direction} by {amount}px",
                duration_ms=round((time.time() - start) * 1000, 2),
            )
        except Exception as e:
            return ExecutionResult(
                success=False,
                action_type="scroll",
                target_description=f"{direction} by {amount}px",
                duration_ms=round((time.time() - start) * 1000, 2),
                error_message=str(e),
            )


class PyAutoGUIExecutionEngine:
    """Executes UI actions via PyAutoGUI's cross-platform OS-level input simulation.

    Used for desktop applications that cannot be automated through browser APIs.
    Coordinates-based: all actions target screen pixel coordinates.

    Safety: FAILSAFE is enabled by default — moving the mouse to the top-left
    corner of the screen aborts all operations immediately. MINIMUM_DURATION
    ensures smooth cursor motion instead of instant jumps.
    """

    def __init__(self, safety_margin_px: int = 10) -> None:
        if not PYAUTOGUI_AVAILABLE:
            raise ImportError(
                "PyAutoGUI is required for desktop automation. "
                "Install with: pip install pyautogui"
            )
        self.safety_margin_px = safety_margin_px

    def click(self, x: float, y: float, clicks: int = 1) -> ExecutionResult:
        """Click at screen coordinates.

        Args:
            x: Horizontal pixel coordinate.
            y: Vertical pixel coordinate.
            clicks: Number of clicks (1=left-click, 2=double-click).
        """
        start = time.time()
        try:
            pyautogui.click(x, y, clicks=clicks)
            return ExecutionResult(
                success=True, action_type="click",
                target_description=f"({x:.0f}, {y:.0f}), {clicks} click(s)",
                duration_ms=round((time.time() - start) * 1000, 2),
            )
        except Exception as e:
            return ExecutionResult(
                success=False, action_type="click",
                target_description=f"({x:.0f}, {y:.0f}), {clicks} click(s)",
                duration_ms=round((time.time() - start) * 1000, 2),
                error_message=str(e),
            )

    def type_text(self, text: str) -> ExecutionResult:
        """Type text at current cursor position.

        The caller must ensure the correct input element is focused before calling.
        """
        start = time.time()
        try:
            pyautogui.typewrite(text, interval=0.03)  # 30ms between characters
            return ExecutionResult(
                success=True, action_type="type",
                target_description=f"text: '{text[:50]}{'...' if len(text)>50 else ''}'",
                duration_ms=round((time.time() - start) * 1000, 2),
            )
        except Exception as e:
            return ExecutionResult(
                success=False, action_type="type",
                target_description=text[:50],
                duration_ms=round((time.time() - start) * 1000, 2),
                error_message=str(e),
            )

    def scroll_at(self, x: float, y: float, clicks: int = 3) -> ExecutionResult:
        """Scroll at screen coordinates (scroll wheel).

        Positive clicks scrolls up, negative scrolls down.
        """
        start = time.time()
        try:
            pyautogui.scroll(clicks, x=x, y=y)
            return ExecutionResult(
                success=True, action_type="scroll",
                target_description=f"({x:.0f}, {y:.0f}), {clicks} scroll clicks",
                duration_ms=round((time.time() - start) * 1000, 2),
            )
        except Exception as e:
            return ExecutionResult(
                success=False, action_type="scroll",
                target_description=f"({x:.0f}, {y:.0f}), {clicks} scroll clicks",
                duration_ms=round((time.time() - start) * 1000, 2),
                error_message=str(e),
            )

    def drag_to(self, x1: float, y1: float, x2: float, y2: float, duration_s: float = 0.5) -> ExecutionResult:
        """Drag from (x1,y1) to (x2,y2) with configurable speed.

        Args:
            x1, y1: Starting coordinates.
            x2, y2: Ending coordinates.
            duration_s: How long the drag motion takes in seconds.
        """
        start = time.time()
        try:
            pyautogui.moveTo(x1, y1, duration=0.1)
            pyautogui.drag(x2 - x1, y2 - y1, duration=duration_s)
            return ExecutionResult(
                success=True, action_type="drag",
                target_description=f"({x1:.0f},{y1:.0f}) → ({x2:.0f},{y2:.0f})",
                duration_ms=round((time.time() - start) * 1000, 2),
            )
        except Exception as e:
            return ExecutionResult(
                success=False, action_type="drag",
                target_description=f"({x1:.0f},{y1:.0f}) → ({x2:.0f},{y2:.0f})",
                duration_ms=round((time.time() - start) * 1000, 2),
                error_message=str(e),
            )

    def hotkey(self, *keys: str) -> ExecutionResult:
        """Press a keyboard shortcut (e.g., Ctrl+C, Alt+F4)."""
        start = time.time()
        try:
            pyautogui.hotkey(*keys)
            return ExecutionResult(
                success=True, action_type="key_press",
                target_description=f"hotkey: {'+'.join(keys)}",
                duration_ms=round((time.time() - start) * 1000, 2),
            )
        except Exception as e:
            return ExecutionResult(
                success=False, action_type="key_press",
                target_description=f"hotkey: {'+'.join(keys)}",
                duration_ms=round((time.time() - start) * 1000, 2),
                error_message=str(e),
            )
```

**BAD vs GOOD: Execution Safety**

```python
# ❌ BAD — No safety margin on clicks; a 5px coordinate drift causes wrong element interaction
pyautogui.click(x=450, y=300)  # Might hit adjacent button

# ✅ GOOD — Coordinates centered in the detected element with explicit safety bounds
element_center = ui_element.center  # (452.5, 301.2)
action = engine.click(element_center[0], element_center[1])

# ❌ BAD — No visibility check before typing into a field
engine.type_into("input[name='email']", user_email)

# ✅ GOOD — Validates the target element is visible and enabled before executing
element = page.locator(f"input[name='{field_name}']")
if not element.is_visible() or element.is_disabled():
    raise ActionError(f"Cannot type into '{field_name}': not actionable on screen")
engine.type_into(field_selector, text)
```

### Pattern 4: Error Recovery for Unexpected Dialogs

GUI automation fails when the actual screen state diverges from expectations — unexpected popups, loading spinners, permission prompts, CAPTCHAs, or connection errors. A robust recovery system classifies what went wrong and applies targeted remedi

…(truncated)