hybridlabor-api Desktop automation protocol (windows-computer-use-mcp)
Tool selection guide
| Goal | Tool | Example |
|---|---|---|
| Click a known button | automation_elements(click, title="Save") |
fast, needs exact title |
| Click by intent | automation_smart(click="the Save button") |
slower, finds across all windows |
| Complex multi-step | automation_mission(run="save file as PDF") |
autonomous loop with retry |
| Record + replay | automation_macro(record) → stop → replay |
repeatable sequences |
| Wait for event | automation_watch(start, "window_appears", "Update") |
event-driven triggers |
| Explore desktop | automation_smart(discover) |
get all running apps |
| Extract + chain | automation_mission(workflow, actions, store_as, dollar-ref) |
cross-app data pipeline |
Focus rules
- After automation starts, do not Alt+Tab or touch the mouse
- The target window must stay in foreground during click/type sequences
- If HITL approval appears, approve it and let the target window regain focus
- Prefer reading logs between paused steps or on a second monitor
Safety
- Read docs/SAFETY.md before production use
- Pair with virtualization-mcp for sandbox/VM isolation when running untrusted automation
- Opt-in features (face, keylogger) are off by default — enable only when needed
Mission design (automation_mission)
For run operation: provide a clear, concrete goal. The server decomposes via LLM sampling.
For workflow operation: provide explicit steps with tool, params, label, and optional store_as/dollar-ref for data chaining.
For self-healing: set app_path on steps so the mission can re-launch crashed apps.