Browser Automation

Browser Automation

Description

Comprehensive operational playbook for EloPhanto's 47 browser tools — strategies, workflows, decision trees, and patterns for reliable web automation across all task types.

Triggers

• browser
• web page
• website
• navigate
• click
• scrape
• fill form
• login
• web automation
• extract data
• screenshot
• download
• monitor
• web scraping
• form submission
• checkout
• booking
• search the web

Instructions

1. The Evidence Gating Rule (Non-Negotiable)

After ANY state-changing action, you MUST observe before your next action.

State-changing actions: browser_click, browser_click_text, browser_click_batch, browser_click_at, browser_type, browser_type_text, browser_press_key, browser_select_option, browser_drag_drop, browser_drag_solve, browser_drag_brute_force, browser_navigate, browser_go_back, browser_scroll, browser_scroll_container, browser_eval (when it modifies DOM), browser_inject

Observation tools (pick the right one):

• browser_get_elements — when you need to interact with what changed (clicking, typing next)
• browser_extract — when you need to read text content that appeared
• browser_screenshot — when you need visual confirmation or the page is complex
• browser_read_semantic — when the page is long/dense and you need a structured overview

Anti-pattern: browser_click → browser_click → browser_type (two actions, no observation) Correct: browser_click → browser_get_elements → browser_type → browser_extract

2. Task Type Strategy Selection

Choose your approach based on what you're doing:

Data Extraction (scraping, reading content)

1. browser_navigate to the page
2. browser_read_semantic for a structured overview of the content
3. browser_extract for full text if you need everything
4. browser_get_html if you need raw HTML, hidden attributes, or structured data
5. For tables/lists, prefer browser_extract over browser_get_html

Form Filling (registration, checkout, data entry)

1. browser_navigate to the form page
2. browser_get_elements to map ALL form fields (inputs, selects, checkboxes)
3. Fill fields in order using browser_type (by element index)
4. For dropdowns: browser_select_option
5. For checkboxes/radios: browser_select_option
6. Observe after each field to catch validation errors
7. Submit with browser_click_text on the submit button, or browser_press_key Enter
8. Observe the result page to confirm success

Multi-Page Navigation (following links, paginated content)

1. Start with browser_navigate
2. Use browser_click_text for clearly labeled links ("Next", "Page 2")
3. Use browser_click with index for links without clear text
4. After each navigation, observe before proceeding
5. For pagination, extract data from each page before navigating to the next
6. Track your position — know which page you're on

Search and Find (looking for specific content)

1. browser_navigate to the site
2. browser_get_elements to find the search input
3. browser_type the query, set enter=true to submit
4. Observe the results with browser_extract or browser_read_semantic
5. If results are paginated, follow the multi-page pattern

Monitoring (checking status, watching for changes)

1. browser_navigate to the target page
2. browser_extract or browser_read_semantic to capture current state
3. For repeated checks, use the same observation tool each time
4. Compare results between checks to detect changes
5. Use browser_get_network to monitor API calls if the page loads data dynamically

Interactive Applications (dashboards, SPAs, web apps)

1. browser_navigate — wait for the app shell to load
2. browser_wait_for_selector for the main content area (SPAs often load asynchronously)
3. browser_read_semantic to understand the app layout
4. browser_get_elements to discover interactive controls
5. For modals/dialogs: browser_scroll_container to scroll within them
6. For tabs within the app: browser_click_text on tab labels

3. Tool Selection Decision Tree

"I need to reach another page on the site"

• The page shows a link or menu item for it → browser_click_text (preferred) or browser_click
• I know the URL from the user, a page, or a search result → browser_navigate
• Nothing visible leads there → web_search for the page, then navigate to the result
• NEVER type a guessed path (/login, /offers, /promotions, /usa/...). A guessed URL 404s or lands on the wrong page and hides what the site really shows. The browser drives the site the way a person does: find the control, click it.

"I need to click something"

• I know the visible text → browser_click_text (preferred)
• I know the element index → browser_click
• I need to click at exact coordinates (canvas/map) → browser_click_at
• I need to click many things fast → browser_click_batch

"I need to type something"

• Into a specific input field → browser_type (with element index)
• Focus is already set → browser_type_text
• I need to press a key (Enter, Tab, Escape) → browser_press_key
• I need to select from a dropdown → browser_select_option

"I need to read the page"

• I want the text content → browser_extract
• I want a structured overview of a long page → browser_read_semantic
• I want the raw HTML → browser_get_html
• I want interactive elements with indices → browser_get_elements
• I want metadata (title, description, OG tags) → browser_get_meta
• I want a visual snapshot → browser_screenshot

"I need to inspect deeply"

• Full audit (DOM + JS + storage + cookies + meta) → browser_full_audit (one call instead of five)
• Hidden data attributes → browser_deep_inspect
• Page scripts and inline JS → browser_read_scripts
• Search for specific text in DOM → browser_dom_search
• Find hidden codes after interactions → browser_extract_hidden_code

"I need to wait for something"

• Wait for a specific element to appear → browser_wait_for_selector (preferred)
• Wait a fixed time → browser_wait (last resort)

"I need to debug"

• Console errors → browser_get_console
• Network requests/responses → browser_get_network
• Specific response body → browser_get_response_body
• Storage state → browser_get_storage
• Cookies → browser_get_cookies

4. Common Workflows

Login Flow (Full)

1. browser_navigate(url="https://example.com")
2. browser_get_elements → check if login form visible
   - If NO login form: already authenticated, proceed with task
   - If login form visible: continue
3. vault_lookup(service="example.com") → check stored credentials
   - If found: use them
   - If not found: ask user for credentials
4. browser_type(index=<email_field>, text="user@example.com")
5. browser_type(index=<password_field>, text="password123")
6. browser_click_text(text="Sign In") or browser_click_text(text="Log In")
7. browser_get_elements → verify login succeeded (no login form, user menu visible)
8. If 2FA/MFA prompt appears: ask user for the code, then browser_type it

Data Extraction with Pagination

1. browser_navigate(url="https://example.com/results")
2. browser_extract → capture page 1 data
3. browser_get_elements → find "Next" button
4. Loop:
   a. browser_click_text(text="Next")
   b. browser_wait_for_selector(selector=".results-loaded")
   c. browser_extract → capture page N data
   d. browser_get_elements → check if "Next" still exists
   e. If no Next button: done

Form with Validation

1. browser_navigate → load the form
2. browser_get_elements → map all fields
3. For each field:
   a. browser_type(index=N, text=value)
   b. browser_press_key(key="Tab") → trigger validation
   c. browser_get_elements → check for error messages
   d. If error: fix the value and retry
4. browser_click_text(text="Submit")
5. browser_extract → read success/error message

Viewing Local HTML Files

NEVER use file:// URLs — they do not work in the automated browser.
Instead, start a local HTTP server first:

1. shell_execute(command="cd /path/to/dir && python3 -m http.server 8080 &")
2. browser_navigate(url="http://localhost:8080/index.html")
3. When done, shell_execute(command="kill $(lsof -t -i:8080)") to stop the server

Download a File

1. browser_navigate → go to the download page
2. browser_click_text(text="Download") or browser_click on the download link
3. browser_wait(ms=3000) → wait for download to start
4. The file lands in Chrome's default download directory

Tab Management for Multi-Site Tasks

1. browser_navigate(url="https://site1.com") → opens in current tab
2. browser_extract → get data from site 1
3. browser_new_tab(url="https://site2.com") → opens site 2 in new tab
4. browser_extract → get data from site 2
5. browser_switch_tab(index=0) → go back to site 1
6. browser_close_tab(index=1) → close site 2 when done

5. Handling Tricky Situations

Modals and Dialogs

• Use browser_get_elements — modal elements appear with high indices
• If the modal has its own scrollable area: browser_scroll_container
• To close: browser_press_key(key="Escape") or browser_click_text(text="Close")
• If overlay blocks clicks: browser_click_at on the overlay's close button

Infinite Scroll Pages

• browser_scroll(direction="down") to load more content
• browser_extract after each scroll to capture new content
• Repeat until content stops changing or you have enough data
• Track the page height via browser_eval to detect when no new content loads

Single Page Applications (React, Vue, Angular)

• The URL may not change on navigation — use browser_wait_for_selector instead of waiting for navigation
• Content loads asynchronously — always wait for data to appear
• Use browser_read_semantic to understand the current view
• React apps: browser_get_console may show useful state info

Cookie Consent Banners

• browser_click_text(text="Accept") or browser_click_text(text="Accept All")
• If that doesn't work: browser_get_elements to find the consent button
• Some banners need browser_scroll_container first

CAPTCHA / Challenge Pages

• Do NOT try to solve CAPTCHAs programmatically
• Report to the user: "I encountered a CAPTCHA on [url]. Please solve it manually, then tell me to continue."
• If using profile mode, the user's browser may have CAPTCHA bypass tokens

Iframes

• browser_get_html to see iframe src URLs
• browser_navigate to the iframe URL directly if you need its content
• Alternatively, browser_eval to access iframe content via JS (if same-origin)

Dynamic Content (AJAX, WebSockets)

• browser_get_network to see what API calls the page makes
• browser_get_response_body to read API response data directly
• Often more reliable than scraping the rendered DOM
• browser_get_console may reveal real-time data updates

6. Performance Patterns

Minimize Tool Calls

• Use browser_full_audit instead of calling browser_get_storage + browser_get_cookies + browser_get_meta + browser_deep_inspect + browser_read_scripts separately
• Use browser_read_semantic instead of browser_extract when you need structure, not raw text
• Use browser_click_batch when clicking multiple elements

Prefer Selectors Over Fixed Waits

• browser_wait_for_selector(selector=".content-loaded") → proceeds as soon as ready
• browser_wait(ms=5000) → always waits the full duration even if ready sooner

Use the Right Observation Tool

• Need element indices for next action? → browser_get_elements
• Need to read text content? → browser_extract
• Need structured overview of complex page? → browser_read_semantic
• Need visual confirmation? → browser_screenshot (most expensive, use when needed)

7. Debugging Strategies

When something isn't working:

1. Take a screenshot — browser_screenshot gives you visual context
2. Check console — browser_get_console for JavaScript errors
3. Check network — browser_get_network for failed API calls or redirects
4. Inspect the element — browser_inspect_element or browser_get_element_html for the specific element
5. Search the DOM — browser_dom_search for text you expect to find
6. Full audit — browser_full_audit for a comprehensive state dump

8. Security Awareness

• The browser uses the user's REAL Chrome profile with real sessions and real data
• Never screenshot or extract content from banking/medical/sensitive sites unless explicitly asked
• Credentials from vault_lookup are typed directly into forms — never include them in your text responses
• browser_get_cookies and browser_get_storage can contain session tokens — handle with care
• If a page asks you to download or run something suspicious, stop and ask the user

Verify

• The intended other agent / tool / channel actually received the message; an ack, message ID, or response payload is captured
• Identity, scopes, and permissions used by the call were the minimum required; over-permissioned tokens are called out
• Failure handling was exercised: at least one retry/timeout/permission-denied path is shown to behave as designed
• Hand-off context passed to the next actor is complete enough that the receiver could act without a follow-up question
• Any state mutated (config, memory, queue, file) is listed with before/after values, not just 'updated'
• Sensitive material (keys, tokens, PII) was redacted from logs/transcripts shared in the verification evidence

Notes

The browser launches on first use (not at startup) and uses the user's real Chrome profile when in profile mode. This means existing logins, cookies, extensions, and bookmarks are all available. The browser bridge communicates via JSON-RPC over stdin/stdout with a Node.js subprocess running Playwright with stealth plugins for anti-detection.

Always-on failure modes (also in system prompt)

These survive even if this skill was only loaded as a summary excerpt:

1. Task restart — After an interrupted prior task, treat browser state in history as STALE. Navigate deliberately for the CURRENT user message.
2. X/Twitter "Post" — Never browser_click_text('Post') to submit; it hits the sidebar. Click the composer submit button by index. Replies use browser_click_text('Reply', exact=True).
3. Signup email — Always identity_status email; never invent an inbox.
4. Evidence gating — Observe after every state-changing action.