Browser Automation

Browser Automation

Overview

Browser Use Cloud provides "dumb hands" for the AI investigator. The orchestrator (Opus) sends natural language instructions, Browser Use executes them in a cloud browser, and returns extracted page text. The user can watch via a live iframe.

API

Base URL: https://api.browser-use.com/api/v3

Auth: X-Browser-Use-API-Key: {BROWSER_USE_API_KEY}

Endpoints Used

Session Statuses (v3)

Task Execution

// convex/tools/browserUse.ts — runTask
// v3: session creation and task execution are combined
const body: Record<string, unknown> = {
  task: "Go to imginn.com/johndoe and describe what you see",
  keepAlive: true,
};
if (existingSessionId) {
  body.sessionId = existingSessionId;  // reuses session
}

const createRes = await fetch(`${API}/sessions`, {
  method: "POST",
  headers: getHeaders(),
  body: JSON.stringify(body),
});
const created = await createRes.json();
const sessionId = created.id;

// Poll GET /sessions/{id} until status is "idle" (finished)
// Adaptive polling: 1s for first 10 checks, then 2s after
// Max 200 attempts (~6 minutes)

Session Reuse

Before reusing a session, waitForSessionIdle() polls the session status for up to 60s:

• If idle → reuse by passing sessionId in the POST body
• If dead (stopped, timed_out, error) → create a fresh session
• If running/created → wait up to 60s, then create fresh

Graceful Error Handling

Terminal states return structured results with recovery hints instead of throwing:

// Instead of: throw new Error("Browser Use task timed out")
// Returns:
{
  output: "Browser timed out. RECOVERY: Use web_search instead.",
  sessionId,
  liveUrl,
  status: "timed_out",
}

This lets the orchestrator pass the recovery hint to Opus, which can then switch to web_search.

Browser Budget

The orchestrator limits browser actions to MAX_BROWSER_ACTIONS (6) per investigation:

• Tracked via browserActionsUsed counter across steps
• When limit reached, browser_action is removed from available tools
• Step context warns: [Browser limit reached. Use web_search for all remaining lookups.]

Integration with Orchestrator

1. startInvestigation checks maigret health, builds initial context, starts the step loop
2. First browser_action call creates the session; sessionId and liveUrl stored on investigation
3. Subsequent browser_action calls reuse the session via waitForSessionIdle
4. Frontend picks up browserLiveUrl and renders it in BrowserView iframe
5. On investigation completion/failure, cleanupBrowserSession stops the session

Frontend: BrowserView Component

src/components/BrowserView.tsx

• Empty state (planning): globe icon + "Waiting for investigation to start..."
• Connecting state (investigating, no URL yet): green pulse + "Connecting to browser..."
• Active state: URL bar (green dot + "LIVE" badge) + full-height iframe

Convex Implementation

All functions are internalAction — only callable by the orchestrator.

Gotchas

• Session reuse: waitForSessionIdle ensures the session is ready before sending a new task
• Extreme mode: Sets model: "bu-2-0" for +12% accuracy
• iframe sandbox: allow-same-origin allow-scripts allow-forms allow-popups needed for Browser Use player
• Retry on 5xx/429: runTask retries up to 2 times with 3s delay for server errors
• 404 on stop: stopSession treats 404 as success — session may already be gone
• Error recovery: Timeouts and errors return structured output with RECOVERY: hints, not exceptions