ddalcu Web Pilot
Tools
| Tool | Purpose |
|---|---|
sandbox_browser_observe |
Text snapshot: URL, title, headings, textExcerpt, elements with refs |
sandbox_browser_navigate |
Go to URL, wait for ready, return snapshot with textExcerpt |
sandbox_browser_click |
Click by ref (e.g. "e3") or visible text |
sandbox_browser_type |
Type into input by ref (e.g. "e5") |
sandbox_browser_content |
Page as markdown (optional CSS selector) |
sandbox_browser_evaluate |
Run JS, reports side effects |
sandbox_browser_screenshot |
PNG screenshot (expensive, last resort) |
sandbox_web_search |
Search DuckDuckGo for URLs |
Workflow: Observe → Act → Observe
- Navigate → returns snapshot with element refs
- Act using refs from snapshot:
click({ ref: "e3" })ortype({ ref: "e5", text: "hello" }) - Observe → verify action worked
- Repeat. Screenshot only if text can't answer your question.
Refs
Observe output lists elements like:
- button "Submit" [ref=e1]
- textbox [ref=e2] name=email placeholder="Enter email"
- link "Home" [ref=e3]
Use refs in click/type: { "ref": "e1" }. Refs update on each observe — always use latest.
Data Extraction Strategy
- Check textExcerpt first — observe/navigate returns the first 2000 chars of visible text. For many tasks this is enough.
- Use content(selector) over content() — target a specific section (e.g.
{ selector: "main" }or{ selector: ".repo-list" }) to avoid huge dumps. - Use evaluate() for structured extraction —
evaluate({ expression: "..." })with JS that returns exactly the data you need (e.g.[...document.querySelectorAll('.repo')].map(...)) - Prefer web_fetch for read-only pages — if you just need page content without interaction,
sandbox_web_fetchis cheaper than navigating. - Never repeat a failed approach — if a tool call returns nothing useful, switch to a different strategy immediately.
Patterns
- Click by text:
{ "text": "Accept cookies" }(when ref unavailable) - Scroll: evaluate
window.scrollBy(0, 500)then observe - Dropdown: evaluate with value set + change event dispatch