Automation Operations (Desktop & Browser Automation)
ACTION REQUIRED (execute immediately after reading)
NOW: confirm whether the current task falls within this skill's scope
NOW: read ../tool-index.md, verify tool availability and actual paths
NEXT: when tools are missing, invoke bootstrap; do not guess paths
ACT: enter step 1 of the "Workflow" and execute; do not stop at the confirmation stage
Scope
Use this skill when the task falls into one of these scenarios:
Browser Scenarios (Playwright / agent-browser)
- Opening web pages and operating page elements (clicking, filling forms, submitting)
- Crawling page content or taking screenshots
- Automating login flows
- Interacting with web pages during penetration testing (submitting payloads, triggering XSS)
- Automated handling of CAPTCHA pages
- Bulk form submission
Desktop Application Scenarios (OpenReverse)
- Operating Windows desktop applications (IDA Pro, x64dbg, Wireshark, etc.)
- Vision-driven interaction needed (CUA mode)
- Structured UI operations needed (UIA mode)
- Network traffic observation of desktop applications (built-in mitmproxy)
- Automated GUI operation of reverse engineering tools
- Black-box testing of desktop software
Division of Labor with Other Tools
| Scenario |
Use |
| Operating web pages (inside the browser) |
Playwright / agent-browser |
| Operating desktop applications (Windows GUI) |
OpenReverse |
| Traffic capture analysis, HTTP request capture |
anything-analyzer or OpenReverse network lane |
| JS breakpoints, hooks, CDP debugging |
jshookmcp |
| Locating signature algorithms, environment-patched reproduction |
js-reverse |
Simple rule:
- Target is a web page → Playwright
- Target is a Windows desktop application → OpenReverse
- Both needed → combine them
Part 1: Browser Automation (Playwright / agent-browser)
Core Workflow
# 1. Open a page
agent-browser open <url>
# 2. Get interactable elements (returns @e1, @e2... references)
agent-browser snapshot -i
# 3. Operate elements via references
agent-browser click @e1
agent-browser fill @e2 "text"
# 4. Close when done
agent-browser close
Command Reference
# Navigation
agent-browser open <url>
agent-browser close
# Page snapshot
agent-browser snapshot # full accessibility tree
agent-browser snapshot -i # interactable elements only (recommended)
# Interaction
agent-browser click @e1
agent-browser fill @e2 "text"
agent-browser type @e2 "text"
agent-browser press Enter
agent-browser scroll down 500
# Get information
agent-browser get text @e1
agent-browser get title
agent-browser get url
# Waiting
agent-browser wait @e1
agent-browser wait 2000
agent-browser wait --load networkidle
Notes
- You must run
agent-browser close, otherwise processes leak
- Snapshot before operating; do not guess element references
- After submitting a form, use
wait --load networkidle to let the page settle
Part 2: Desktop Application Automation (OpenReverse)
Overview
OpenReverse is a desktop interaction and evidence-collection framework for AI agents, supporting:
- UIA mode: Windows UI Automation, structured desktop control operations
- CUA mode: vision-driven interaction (Computer Use Agent), suited for complex GUIs
- Network observation: built-in mitmproxy proxy + local capture
Interaction Mode Selection
| Mode |
Suited Scenario |
Underlying |
| UIA |
Target app has standard Windows controls (buttons, text boxes, lists) |
Windows UI Automation API |
| CUA |
Target app has a complex UI or non-standard controls (IDA's disassembly view, custom-rendered interfaces) |
Vision recognition + mouse/keyboard |
Network Observation Modes
| Mode |
Suited Scenario |
| Proxy Lane |
Target app can be configured with a proxy (recommended) |
| Local Lane |
Target app cannot go through a proxy; local capture needed |
Installation and Configuration
# 1. Clone the project
git clone https://github.com/zhexulong/openreverse.git
cd openreverse
# 2. Install dependencies
npm install
# 3. Hook into agent hosts (Claude Code / Codex / Zed)
npm run init:agents -- --target=all /path/to/project
# 4. Install the CUA runtime (if vision-driven mode is needed)
npm run install:cua-runtime
npm run doctor:cua-runtime
# 5. Install network observation dependencies (if traffic capture is needed)
npm run install:mitmproxy
npm run doctor:network
Common Combinations
| Need |
Configuration |
| Only operate desktop applications |
UIA or CUA, no network lane |
| Operate desktop apps + capture traffic |
UIA/CUA + proxy lane |
| Operate desktop apps + local capture |
UIA/CUA + local lane |
Reverse Engineering Scenario Examples
Scenario: automating IDA Pro for batch analysis
1. Open IDA Pro via OpenReverse CUA mode
2. Automatically load the target binary
3. Wait for analysis to finish
4. Export the function list via UI operations
5. Meanwhile use the network lane to observe IDA's network behavior (e.g., Lumina requests)
Scenario: automating x64dbg debugging
1. Launch x64dbg via OpenReverse UIA mode
2. Load the target program
3. Set breakpoints
4. Run and observe register/memory changes
5. Save screenshots as evidence
On-Demand Bootstrap
Automation Capability Boundaries
| Tool |
Auto-installable |
Install Method |
Notes |
| Playwright |
✓ |
npm + npx playwright install |
Browser automation engine |
| agent-browser CLI |
✓ |
npm install -g agent-browser |
Browser operation CLI |
| Node.js |
✓ |
winget |
Prerequisite dependency |
| OpenReverse |
✗ |
Manual clone + npm install |
Experimental stage, heavy dependencies |
| mitmproxy |
✗ |
Manual install |
OpenReverse network observation dependency |
Bootstrap Triggers
- Browser operation missing Playwright → auto bootstrap
- Desktop operation needs OpenReverse → guide the user through manual installation (provide full steps)
OpenReverse Manual Installation Guide
If the AI detects that desktop application automation is needed but OpenReverse is not installed:
⚠️ **OpenReverse is needed for desktop application automation**
**Installation steps**:
1. `git clone https://github.com/zhexulong/openreverse.git`
2. `cd openreverse && npm install`
3. `npm run init:agents -- --target=all <your project path>`
4. If vision mode is needed: `npm run install:cua-runtime`
5. If network observation is needed: `npm run install:mitmproxy`
**Verify**: `npm run doctor:cua-runtime` and `npm run doctor:network`
Routing Context
Upstream entry points: skills/SKILL.md (master control), routing.md
Applicable scenarios: any task that requires automating a browser or desktop application
Downstream exits:
- Captured requests need analysis →
anything-analyzer or js-reverse
- JS debugging/hooks needed →
jshookmcp
- Signature algorithm restoration needed →
js-reverse
- Desktop app is a reverse engineering tool →
ida-reverse/
Related sibling modules: js-reverse (after browser operations, JS analysis may be needed), ida-reverse (OpenReverse can automate the IDA GUI)
Task Completion Self-Check (MUST pass before claiming completion)
1---2name: router-reverse-skill-router-browser-automation3description: Unified automation entry point. Covers browser automation (Playwright) and Windows desktop application automation (OpenReverse). Browser scenarios: opening web pages, clicking, filling forms, crawling, screenshots, automated login, pentest page interaction. Desktop scenarios: operating GUI tools like IDA/x64dbg, Windows UI Automation, vision-driven interaction, desktop application traffic capture. Trigger keywords: browser automation, desktop automation, open web page, fill form, crawl, screenshot, automated login, Playwright, agent-browser, headless, OpenReverse, UIA, CUA, desktop operation, Windows automation.4---56# Automation Operations (Desktop & Browser Automation)78## ACTION REQUIRED (execute immediately after reading)9101. `NOW`: confirm whether the current task falls within this skill's scope112. `NOW`: read `../tool-index.md`, verify tool availability and actual paths123. `NEXT`: when tools are missing, invoke bootstrap; do not guess paths134. `ACT`: enter step 1 of the "Workflow" and execute; do not stop at the confirmation stage1415## Scope1617Use this skill when the task falls into one of these scenarios:1819### Browser Scenarios (Playwright / agent-browser)20- Opening web pages and operating page elements (clicking, filling forms, submitting)21- Crawling page content or taking screenshots22- Automating login flows23- Interacting with web pages during penetration testing (submitting payloads, triggering XSS)24- Automated handling of CAPTCHA pages25- Bulk form submission2627### Desktop Application Scenarios (OpenReverse)28- Operating Windows desktop applications (IDA Pro, x64dbg, Wireshark, etc.)29- Vision-driven interaction needed (CUA mode)30- Structured UI operations needed (UIA mode)31- Network traffic observation of desktop applications (built-in mitmproxy)32- Automated GUI operation of reverse engineering tools33- Black-box testing of desktop software3435### Division of Labor with Other Tools3637| Scenario | Use |38|------|--------|39| Operating web pages (inside the browser) | **Playwright / agent-browser** |40| Operating desktop applications (Windows GUI) | **OpenReverse** |41| Traffic capture analysis, HTTP request capture | anything-analyzer or OpenReverse network lane |42| JS breakpoints, hooks, CDP debugging | jshookmcp |43| Locating signature algorithms, environment-patched reproduction | js-reverse |4445Simple rule:46- Target is a web page → Playwright47- Target is a Windows desktop application → OpenReverse48- Both needed → combine them4950---5152## Part 1: Browser Automation (Playwright / agent-browser)5354### Core Workflow5556```bash57# 1. Open a page58agent-browser open <url>5960# 2. Get interactable elements (returns @e1, @e2... references)61agent-browser snapshot -i6263# 3. Operate elements via references64agent-browser click @e165agent-browser fill @e2 "text"6667# 4. Close when done68agent-browser close69```7071### Command Reference7273```bash74# Navigation75agent-browser open <url>76agent-browser close7778# Page snapshot79agent-browser snapshot # full accessibility tree80agent-browser snapshot -i # interactable elements only (recommended)8182# Interaction83agent-browser click @e184agent-browser fill @e2 "text"85agent-browser type @e2 "text"86agent-browser press Enter87agent-browser scroll down 5008889# Get information90agent-browser get text @e191agent-browser get title92agent-browser get url9394# Waiting95agent-browser wait @e196agent-browser wait 200097agent-browser wait --load networkidle98```99100### Notes101- You must run `agent-browser close`, otherwise processes leak102- Snapshot before operating; do not guess element references103- After submitting a form, use `wait --load networkidle` to let the page settle104105---106107## Part 2: Desktop Application Automation (OpenReverse)108109### Overview110111[OpenReverse](https://github.com/zhexulong/openreverse) is a desktop interaction and evidence-collection framework for AI agents, supporting:112- **UIA mode**: Windows UI Automation, structured desktop control operations113- **CUA mode**: vision-driven interaction (Computer Use Agent), suited for complex GUIs114- **Network observation**: built-in mitmproxy proxy + local capture115116### Interaction Mode Selection117118| Mode | Suited Scenario | Underlying |119|------|---------|------|120| UIA | Target app has standard Windows controls (buttons, text boxes, lists) | Windows UI Automation API |121| CUA | Target app has a complex UI or non-standard controls (IDA's disassembly view, custom-rendered interfaces) | Vision recognition + mouse/keyboard |122123### Network Observation Modes124125| Mode | Suited Scenario |126|------|---------|127| Proxy Lane | Target app can be configured with a proxy (recommended) |128| Local Lane | Target app cannot go through a proxy; local capture needed |129130### Installation and Configuration131132```bash133# 1. Clone the project134git clone https://github.com/zhexulong/openreverse.git135cd openreverse136137# 2. Install dependencies138npm install139140# 3. Hook into agent hosts (Claude Code / Codex / Zed)141npm run init:agents -- --target=all /path/to/project142143# 4. Install the CUA runtime (if vision-driven mode is needed)144npm run install:cua-runtime145npm run doctor:cua-runtime146147# 5. Install network observation dependencies (if traffic capture is needed)148npm run install:mitmproxy149npm run doctor:network150```151152### Common Combinations153154| Need | Configuration |155|------|------|156| Only operate desktop applications | UIA or CUA, no network lane |157| Operate desktop apps + capture traffic | UIA/CUA + proxy lane |158| Operate desktop apps + local capture | UIA/CUA + local lane |159160### Reverse Engineering Scenario Examples161162```text163Scenario: automating IDA Pro for batch analysis1641651. Open IDA Pro via OpenReverse CUA mode1662. Automatically load the target binary1673. Wait for analysis to finish1684. Export the function list via UI operations1695. Meanwhile use the network lane to observe IDA's network behavior (e.g., Lumina requests)170```171172```text173Scenario: automating x64dbg debugging1741751. Launch x64dbg via OpenReverse UIA mode1762. Load the target program1773. Set breakpoints1784. Run and observe register/memory changes1795. Save screenshots as evidence180```181182---183184## On-Demand Bootstrap185186### Automation Capability Boundaries187188| Tool | Auto-installable | Install Method | Notes |189|------|-----------|---------|------|190| Playwright | ✓ | npm + npx playwright install | Browser automation engine |191| agent-browser CLI | ✓ | npm install -g agent-browser | Browser operation CLI |192| Node.js | ✓ | winget | Prerequisite dependency |193| OpenReverse | ✗ | Manual clone + npm install | Experimental stage, heavy dependencies |194| mitmproxy | ✗ | Manual install | OpenReverse network observation dependency |195196### Bootstrap Triggers197198- Browser operation missing Playwright → auto bootstrap199- Desktop operation needs OpenReverse → guide the user through manual installation (provide full steps)200201### OpenReverse Manual Installation Guide202203If the AI detects that desktop application automation is needed but OpenReverse is not installed:204205```markdown206⚠️ **OpenReverse is needed for desktop application automation**207208**Installation steps**:2091. `git clone https://github.com/zhexulong/openreverse.git`2102. `cd openreverse && npm install`2113. `npm run init:agents -- --target=all <your project path>`2124. If vision mode is needed: `npm run install:cua-runtime`2135. If network observation is needed: `npm run install:mitmproxy`214215**Verify**: `npm run doctor:cua-runtime` and `npm run doctor:network`216```217218---219220## Routing Context221222**Upstream entry points**: `skills/SKILL.md` (master control), `routing.md`223**Applicable scenarios**: any task that requires automating a browser or desktop application224**Downstream exits**:225- Captured requests need analysis → `anything-analyzer` or `js-reverse`226- JS debugging/hooks needed → `jshookmcp`227- Signature algorithm restoration needed → `js-reverse`228- Desktop app is a reverse engineering tool → `ida-reverse/`229230**Related sibling modules**: `js-reverse` (after browser operations, JS analysis may be needed), `ida-reverse` (OpenReverse can automate the IDA GUI)231232233## Task Completion Self-Check (MUST pass before claiming completion)234235- [ ] Did I execute every step of the workflow (rather than just reading it)?236- [ ] Did I use real tool paths based on `tool-index`?237- [ ] Did I produce reproducible evidence (commands/scripts/screenshots/reports)?238- [ ] Did I complete and write back the Checklist items required by RULES?