Desktop Computer Automation

Desktop Computer Automation

CRITICAL RULES:

1. Never run midscene commands in the background. Each command must run synchronously so you can read its output (especially screenshots) before deciding the next action.
2. Run only one midscene command at a time. Wait for the previous command to finish, read the screenshot, then decide the next action.
3. Allow enough time for each command to complete. Midscene commands involve AI inference and screen interaction, which can take longer than typical shell commands.
4. Always report task results before finishing.

Control your desktop (macOS, Windows, Linux) using npx @midscene/computer@1. Each CLI command maps directly to an MCP tool -- you (the AI agent) act as the brain, deciding which actions to take based on screenshots.

Prerequisites

Midscene requires models with strong visual grounding capabilities. Configure these environment variables:

MIDSCENE_MODEL_API_KEY="your-api-key"
MIDSCENE_MODEL_NAME="model-name"
MIDSCENE_MODEL_BASE_URL="https://..."
MIDSCENE_MODEL_FAMILY="family-identifier"

Commands

Connect to Desktop

npx @midscene/computer@1 connect
npx @midscene/computer@1 connect --displayId <id>

List Displays

npx @midscene/computer@1 list_displays

Take Screenshot

npx @midscene/computer@1 take_screenshot

Perform Action

Use act to interact with the computer. Describe what you want to do in natural language:

npx @midscene/computer@1 act --prompt "type hello world in the search field and press Enter"
npx @midscene/computer@1 act --prompt "drag the file icon to the Trash"
npx @midscene/computer@1 act --prompt "search for the weather in Shanghai using the Chrome browser, tell me the result"

Disconnect

npx @midscene/computer@1 disconnect

Workflow Pattern

1. Connect to establish a session
2. Health check -- take a screenshot and verify it succeeds, then move the mouse to a random position
3. Launch the target app and take screenshot to see the current state
4. Execute action using act to perform the desired action
5. Disconnect when done
6. Report results -- summarize what was accomplished

Best Practices

1. Always run a health check first after connecting
2. Bring the target app to the foreground before using this skill (e.g., open -a <AppName> on macOS)
3. Be specific about UI elements: Say "the red close button in the top-left corner" instead of "the close button"
4. Describe locations when possible: "the icon in the top-right corner of the menu bar"
5. Never run in background: Every midscene command must run synchronously
6. Check for multiple displays: Use list_displays if an app is not visible
7. Batch related operations into a single act command when possible
8. Set up PATH before running (macOS): export PATH="/usr/sbin:/usr/bin:/bin:/sbin:$PATH"

Troubleshooting

macOS: Accessibility Permission Denied

Open System Settings > Privacy & Security > Accessibility and add your terminal app.

macOS: Xcode Command Line Tools Not Found

xcode-select --install

API Key Not Set

Check .env file contains MIDSCENE_MODEL_API_KEY=<your-key>.

AI Cannot Find the Element

1. Take a screenshot to verify the element is actually visible
2. Use more specific descriptions (include color, position, surrounding text)
3. Ensure the element is not hidden behind another window