# Drive web and app UIs with vision-grounded steps when selectors are brittle or unavailable

> Use Midscene.js when an agent needs screenshot-grounded UI actions and assertions across web, mobile, or desktop surfaces where DOM selectors are fragile, unavailable, or not the right abstraction.

- Skill: `agentskillexchange/drive-web-and-app-uis-with-vision-grounded-steps-when-select` (Agent Skill)
- Install (CLI): `npx skillmds@latest add agentskillexchange/drive-web-and-app-uis-with-vision-grounded-steps-when-select`
- Raw SKILL.md: https://api.skillmd.com/api/skills/agentskillexchange/drive-web-and-app-uis-with-vision-grounded-steps-when-select/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: agentskillexchange (https://skillmd.com/u/agentskillexchange)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/agentskillexchange/drive-web-and-app-uis-with-vision-grounded-steps-when-select

---


# Drive web and app UIs with vision-grounded steps when selectors are brittle or unavailable

Use Midscene.js when an agent needs screenshot-grounded UI actions and assertions across web, mobile, or desktop surfaces where DOM selectors are fragile, unavailable, or not the right abstraction.

## Prerequisites

Midscene.js, Node.js, a supported vision model, and a target automation surface such as Playwright, Puppeteer, Android adb, or iOS WebDriverAgent

## Installation

Requirements and caveats from upstream:
- [midscene-pc-docker](https://github.com/Mofangbao/midscene-pc-docker) - Docker image with Midscene-PC server pre-installed
- [Midscene-Python](https://github.com/Python51888/Midscene-Python) - Python SDK for Midscene automation

Basic usage or getting-started notes:
- Sample Projects: [https://github.com/web-infra-dev/midscene-example](https://github.com/web-infra-dev/midscene-example)

- Source: https://github.com/web-infra-dev/midscene
- Extracted from upstream docs: https://raw.githubusercontent.com/web-infra-dev/midscene/HEAD/README.md

## Documentation

- https://midscenejs.com

## Source

- [Agent Skill Exchange](https://agentskillexchange.com/skills/drive-web-and-app-uis-with-vision-grounded-steps-when-selectors-are-brittle-or-unavailable/)