Scrapeless Scraping Browser

Cloud Browser Automation with scrapeless-browser

README.md

Important: Session Management with --session-id

All browser operation commands support the --session-id parameter to specify which Scrapeless session to use.

Recommended Workflow

# Step 1: Create a session and save the session ID
SESSION_ID=$(scrapeless-scraping-browser new-session --name "workflow" --ttl 1800 --json | jq -r '.taskId')

# Step 2: Use the session ID for all operations
scrapeless-scraping-browser --session-id $SESSION_ID open https://example.com
scrapeless-scraping-browser --session-id $SESSION_ID snapshot -i
scrapeless-scraping-browser --session-id $SESSION_ID click @e1

# Step 3: Close when done
scrapeless-scraping-browser --session-id $SESSION_ID close

Automatic Session Management

If you don't specify --session-id:

1. The CLI will query for running sessions
2. If a running session exists, it will use the latest one
3. If no running session exists, it will create a new one automatically

For production workflows, always use --session-id to ensure consistency.

Authentication Setup

Before using scrapeless-browser, you MUST set up authentication:

# Method 1: Config file (recommended, persistent)
scrapeless-scraping-browser config set apiKey your_api_token_here

# Method 2: Environment variable
export SCRAPELESS_API_KEY=your_api_token_here

# Verify it's set
scrapeless-scraping-browser config get apiKey

Get your API token from https://app.scrapeless.com

Session Management Behavior

The CLI manages Scrapeless sessions with the following behavior:

• Session Creation: First command creates a new Scrapeless session
• Session Persistence: Sessions remain active only while connection is maintained
• Session Termination: Sessions automatically terminate when connection closes
• Reconnection Limitation: Cannot reconnect to terminated sessions

Important: For multi-step workflows, consider using the TypeScript API to maintain persistent connections.

Core Workflow

Every browser automation follows this pattern:

1. Create Session: Create a session and save the session ID
2. Navigate: Use --session-id to navigate to URL
3. Snapshot: Get element refs with --session-id
4. Interact: Use refs to click, fill, select with --session-id
5. Re-snapshot: After navigation or DOM changes, get fresh refs

# Set API token first
scrapeless-scraping-browser config set apiKey your_token

# Create session
SESSION_ID=$(scrapeless-scraping-browser new-session --name "form-fill" --ttl 600 --json | jq -r '.taskId')

# Start automation with session ID
scrapeless-scraping-browser --session-id $SESSION_ID open https://example.com/form
scrapeless-scraping-browser --session-id $SESSION_ID snapshot -i
# Output: @e1 [input type="email"], @e2 [input type="password"], @e3 [button] "Submit"

scrapeless-scraping-browser --session-id $SESSION_ID fill @e1 "user@example.com"
scrapeless-scraping-browser --session-id $SESSION_ID fill @e2 "password123"
scrapeless-scraping-browser --session-id $SESSION_ID click @e3
scrapeless-scraping-browser --session-id $SESSION_ID wait --load networkidle
scrapeless-scraping-browser --session-id $SESSION_ID snapshot -i  # Check result

Command Chaining

Commands can be chained with && in a single shell invocation:

# Chain open + wait + snapshot
scrapeless-scraping-browser open https://example.com && scrapeless-scraping-browser wait --load networkidle && scrapeless-scraping-browser snapshot -i

# Chain multiple interactions
scrapeless-scraping-browser fill @e1 "user@example.com" && scrapeless-scraping-browser fill @e2 "password123" && scrapeless-scraping-browser click @e3

When to chain: Use && when you don't need to read intermediate output. Run commands separately when you need to parse output first (e.g., snapshot to discover refs, then interact).

Essential Commands

Note: All commands below support the optional --session-id <id> parameter.

# Navigation & Session
scrapeless-scraping-browser new-session [options]              # Create new browser session
scrapeless-scraping-browser [--session-id <id>] open <url>     # Navigate to URL
scrapeless-scraping-browser [--session-id <id>] close          # Close browser session
scrapeless-scraping-browser sessions                           # List running sessions
scrapeless-scraping-browser stop <taskId>                      # Stop specific session
scrapeless-scraping-browser stop-all                           # Stop all sessions

Session Creation with Advanced Options

The new-session command supports extensive customization options:

# Basic session creation
scrapeless-scraping-browser new-session --name "my-session" --ttl 1800

# Session with proxy settings
scrapeless-scraping-browser new-session \
  --name "proxy-session" \
  --proxy-country US \
  --proxy-state CA \
  --proxy-city "Los Angeles" \
  --ttl 3600

# Session with custom browser configuration
scrapeless-scraping-browser new-session \
  --name "mobile-session" \
  --platform macOS \
  --screen-width 375 \
  --screen-height 812 \
  --user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36" \
  --timezone "America/Los_Angeles" \
  --languages "en,es"

# Session with recording enabled
scrapeless-scraping-browser new-session \
  --name "recorded-session" \
  --recording true \
  --ttl 7200

Available Options:

• --help: Display help information and all available parameters
• --name <name>: Session name for identification
• --ttl <seconds>: Session timeout in seconds (default: 180)
• --recording <true|false>: Enable session recording
• --proxy-country <code>: Proxy country code (e.g., AU, US, GB, CN, JP)
• --proxy-state <state>: Proxy state/region (e.g., NSW, CA, NY, TX)
• --proxy-city <city>: Proxy city (e.g., sydney, newyork, london, tokyo)
• --user-agent <ua>: Custom user agent string
• --platform <platform>: Platform (Windows, macOS, Linux)
• --screen-width <px>: Screen width in pixels (default: 1920)
• --screen-height <px>: Screen height in pixels (default: 1080)
• --timezone <tz>: Timezone (default: America/New_York)
• --languages <langs>: Comma-separated language codes (default: en)

# Snapshot
scrapeless-scraping-browser [--session-id <id>] snapshot -i             # Interactive elements with refs (recommended)
scrapeless-scraping-browser [--session-id <id>] snapshot -i -C          # Include cursor-interactive elements
scrapeless-scraping-browser [--session-id <id>] snapshot -s "#selector" # Scope to CSS selector

# Interaction (use @refs from snapshot)
scrapeless-scraping-browser [--session-id <id>] click @e1               # Click element
scrapeless-scraping-browser [--session-id <id>] fill @e2 "text"         # Clear and type text
scrapeless-scraping-browser [--session-id <id>] type @e2 "text"         # Type without clearing
scrapeless-scraping-browser [--session-id <id>] press Enter             # Press key
scrapeless-scraping-browser [--session-id <id>] scroll down 500         # Scroll page
scrapeless-scraping-browser [--session-id <id>] scroll down 500 --selector "div.content" # Scroll within element

# Get information
scrapeless-scraping-browser [--session-id <id>] get text @e1            # Get element text
scrapeless-scraping-browser [--session-id <id>] get url                 # Get current URL
scrapeless-scraping-browser [--session-id <id>] get title               # Get page title
scrapeless-scraping-browser [--session-id <id>] screenshot              # Take screenshot
scrapeless-scraping-browser [--session-id <id>] screenshot --full       # Full page screenshot

# Wait
scrapeless-scraping-browser [--session-id <id>] wait @e1                # Wait for element
scrapeless-scraping-browser [--session-id <id>] wait --load networkidle # Wait for network idle
scrapeless-scraping-browser [--session-id <id>] wait --url "**/page"    # Wait for URL pattern
scrapeless-scraping-browser [--session-id <id>] wait 2000               # Wait milliseconds

# Cookies & Storage
scrapeless-scraping-browser [--session-id <id>] cookies                   # Get all cookies
scrapeless-scraping-browser [--session-id <id>] cookies set <name> <val>  # Set cookie
scrapeless-scraping-browser [--session-id <id>] cookies clear             # Clear cookies
scrapeless-scraping-browser [--session-id <id>] storage local             # Get localStorage
scrapeless-scraping-browser [--session-id <id>] storage local set <k> <v> # Set localStorage

# Multi-page
scrapeless-scraping-browser [--session-id <id>] pages                   # List all pages/tabs
scrapeless-scraping-browser [--session-id <id>] page <pageId>           # Switch to page
scrapeless-scraping-browser [--session-id <id>] tab new [url]           # Open new tab
scrapeless-scraping-browser [--session-id <id>] tab close [n]           # Close tab

# Live preview
scrapeless-scraping-browser live [taskId]                              # Get live preview URL

This document covers the most common usage patterns and commands. The Scrapeless Agentic Browser supports many more commands than listed here. To see the full command list, run:

scrapeless-scraping-browser --help

Core Commands Reference

The following core commands are available for browser automation:

Navigation & Session Management

• open <url> - Navigate to URL
• close - Close browser session

Element Interaction

• click <sel> - Click element (or @ref)
• dblclick <sel> - Double-click element
• hover <sel> - Hover over element
• focus <sel> - Focus element
• scrollintoview <sel> - Scroll element into view

Text Input

• type <sel> <text> - Type into element without clearing
• fill <sel> <text> - Clear and fill element with text
• press <key> - Press key (Enter, Tab, Control+a)
• keyboard type <text> - Type text with real keystrokes (no selector required)
• keyboard inserttext <text> - Insert text without key events

Form Controls

• check <sel> - Check checkbox
• uncheck <sel> - Uncheck checkbox
• select <sel> <val...> - Select dropdown option(s)

Drag & Drop / File Operations

• drag <src> <dst> - Drag and drop from source to destination
• upload <sel> <files...> - Upload files to input element
• download <sel> <path> - Download file by clicking element

Scrolling

• scroll <dir> [px] - Scroll page (up/down/left/right) with optional pixel amount

Wait & Timing

• wait <sel|ms> - Wait for element to appear or wait specified milliseconds

Capture & Export

• screenshot [path] - Take screenshot (optionally save to path)
• pdf <path> - Save current page as PDF

Advanced Operations

• snapshot - Get accessibility tree with refs (optimized for AI agents)
• eval <js> - Execute custom JavaScript code

Common Patterns

Form Submission

scrapeless-scraping-browser config set apiKey your_token

scrapeless-scraping-browser open https://example.com/signup
scrapeless-scraping-browser snapshot -i
scrapeless-scraping-browser fill @e1 "Jane Doe"
scrapeless-scraping-browser fill @e2 "jane@example.com"
scrapeless-scraping-browser click @e3
scrapeless-scraping-browser wait --load networkidle

Data Extraction

scrapeless-scraping-browser config set apiKey your_token

scrapeless-scraping-browser open https://example.com/products
scrapeless-scraping-browser snapshot -i --json
scrapeless-scraping-browser get text @e5 --json

Common Session Configuration Scenarios

Geographic Content Testing

# Access content from different regions
SESSION_ID=$(scrapeless-scraping-browser new-session \
  --name "geo-test" \
  --proxy-country AU \
  --proxy-city sydney \
  --timezone "Australia/Sydney" \
  --languages "en-AU,en" \
  --json | jq -r '.taskId')

scrapeless-scraping-browser --session-id $SESSION_ID open https://example.com

High-Resolution Desktop Testing

# Test on high-resolution displays
SESSION_ID=$(scrapeless-scraping-browser new-session \
  --name "desktop-4k" \
  --platform macOS \
  --screen-width 3840 \
  --screen-height 2160 \
  --json | jq -r '.taskId')

scrapeless-scraping-browser --session-id $SESSION_ID open https://example.com

Session Recording for Debugging

# Enable recording for troubleshooting
SESSION_ID=$(scrapeless-scraping-browser new-session \
  --name "debug-session" \
  --recording true \
  --ttl 7200 \
  --json | jq -r '.taskId')

scrapeless-scraping-browser --session-id $SESSION_ID open https://example.com
# Session recording will be available for review

Session Persistence

Important: Scrapeless sessions terminate when connections close. For persistent workflows, use the TypeScript API:

scrapeless-scraping-browser config set apiKey your_token

# Create a session for login
scrapeless-scraping-browser create --name "login-session" --ttl 1800
scrapeless-scraping-browser open https://app.example.com/login
scrapeless-scraping-browser snapshot -i
scrapeless-scraping-browser fill @e1 "username"
scrapeless-scraping-browser fill @e2 "password"
scrapeless-scraping-browser click @e3
scrapeless-scraping-browser wait --url "**/dashboard"

# For subsequent operations, create a new session
# (Cannot reuse previous session due to connection termination)
scrapeless-scraping-browser create --name "dashboard-session" --ttl 1800
scrapeless-scraping-browser open https://app.example.com/dashboard

Better Alternative: Use TypeScript API for multi-step workflows:

import { BrowserManager } from './dist/browser.js';

const manager = new BrowserManager();
await manager.launch({ id: 'persistent-workflow', action: 'launch' });

const page = manager.getPage();
await page.goto('https://app.example.com/login');
// Login persists throughout the session
await page.fill('#username', 'user');
await page.fill('#password', 'pass');
await page.click('#login');
await page.waitForURL('**/dashboard');
await page.goto('https://app.example.com/profile');
// Session and cookies maintained

await manager.close();

Using Proxies

scrapeless-scraping-browser config set apiKey your_token

# Use residential proxy from specific country
scrapeless-scraping-browser config set proxyCountry US
scrapeless-scraping-browser open https://example.com

# Use custom proxy
scrapeless-scraping-browser config set proxyUrl "http://user:pass@proxy.com:8080"
scrapeless-scraping-browser open https://example.com

# Use proxy with state and city (v2 API)
scrapeless-scraping-browser config set proxyCountry US
scrapeless-scraping-browser config set proxyState CA
scrapeless-scraping-browser config set proxyCity "Los Angeles"
scrapeless-scraping-browser open https://example.com

Browser Fingerprinting

scrapeless-scraping-browser config set apiKey your_token

# Set browser fingerprint to avoid detection
scrapeless-scraping-browser config set fingerprint chrome
scrapeless-scraping-browser open https://example.com

# Customize browser fingerprint details
scrapeless-scraping-browser config set userAgent "your-agent"
scrapeless-scraping-browser config set platform macOS
scrapeless-scraping-browser config set screenWidth 375
scrapeless-scraping-browser config set screenHeight 812
scrapeless-scraping-browser config set timezone "Asia/Shanghai"
scrapeless-scraping-browser config set languages "zh-CN,en"
scrapeless-scraping-browser open https://example.com

Session Recording

scrapeless-scraping-browser config set apiKey your_token

# Enable session recording for debugging
scrapeless-scraping-browser config set sessionRecording true
scrapeless-scraping-browser open https://example.com
# ... perform actions ...
scrapeless-scraping-browser close
# Recording will be available in Scrapeless dashboard

Multiple Sessions

Note: Due to session termination behavior, the --session-id parameter has limitations. For reliable multi-session workflows, create separate sessions for each task:

scrapeless-scraping-browser config set apiKey your_token

# Create first session for task A
scrapeless-scraping-browser create --name "task-a" --ttl 1800
scrapeless-scraping-browser open https://site-a.com
# Complete task A operations...

# Create second session for task B  
scrapeless-scraping-browser create --name "task-b" --ttl 1800
scrapeless-scraping-browser open https://site-b.com
# Complete task B operations...

# List all running sessions
scrapeless-scraping-browser sessions

# Stop specific session
scrapeless-scraping-browser stop <taskId>

# Stop all sessions
scrapeless-scraping-browser stop-all

Alternative: For complex multi-session workflows, use the TypeScript API which supports persistent connections.

Configuration File

Configuration is managed via the config command. All settings are stored in ~/.scrapeless/config.json.

Priority: Config file > Environment variable (only SCRAPELESS_API_KEY supports env var)

Available configuration options:

• apiKey - Your Scrapeless API token (required)
• apiVersion - API version (v1 or v2, default: v2)
• sessionTtl - Session timeout in seconds
• sessionName - Session name for identification
• sessionRecording - Enable session recording (true/false)
• proxyUrl - Custom proxy URL
• proxyCountry - Proxy country code
• proxyState - Proxy state/province
• proxyCity - Proxy city
• fingerprint - Browser fingerprint
• debug - Enable debug logging

Agent Mode (JSON Output)

Use --json for machine-readable output:

scrapeless-scraping-browser snapshot -i --json
scrapeless-scraping-browser get text @e1 --json
scrapeless-scraping-browser is visible @e2 --json

Ref Lifecycle (Important)

Refs (@e1, @e2, etc.) are invalidated when the page changes. Always re-snapshot after:

• Clicking links or buttons that navigate
• Form submissions
• Dynamic content loading

scrapeless-scraping-browser click @e5              # Navigates to new page
scrapeless-scraping-browser snapshot -i            # MUST re-snapshot
scrapeless-scraping-browser click @e1              # Use new refs

Session Management

Important Session Behavior

Critical: Scrapeless sessions have specific connection requirements:

• ✅ Sessions work perfectly with persistent connections
• ❌ Sessions automatically terminate when the connection is closed
• ❌ Reconnecting to a terminated session will fail

Recommended Usage Patterns

For Single Operations

# Create and use a session for a single task
scrapeless-scraping-browser create --name "single-task" --ttl 600
scrapeless-scraping-browser open https://example.com
scrapeless-scraping-browser screenshot

For Multi-Step Operations

For complex workflows requiring multiple steps, use the TypeScript API instead of CLI:

import { BrowserManager } from './dist/browser.js';

const manager = new BrowserManager();
await manager.launch({ id: 'workflow', action: 'launch' });

const page = manager.getPage();
await page.goto('https://example.com');
await page.screenshot({ path: 'step1.png' });
await page.goto('https://another-site.com');
await page.screenshot({ path: 'step2.png' });

await manager.close();

Session ID Parameter Limitations

The --session-id parameter has limitations due to Scrapeless session behavior:

# This will fail if the session connection was previously closed
scrapeless-scraping-browser --session-id <session-id> open https://example.com
# Error: Session has been terminated and cannot be reconnected

Workaround: Create new sessions for each workflow instead of reusing session IDs.

Session Management Commands

Always close sessions when done to avoid leaked resources:

scrapeless-scraping-browser close                    # Close current session
scrapeless-scraping-browser stop <taskId>            # Stop specific session by ID
scrapeless-scraping-browser stop-all                 # Stop all sessions

Check running sessions:

scrapeless-scraping-browser sessions
# Returns: sessionId, createdAt, status, sessionName

Live Preview

Get a real-time preview of your browser session via WebSocket:

# Get live preview URL for current session
scrapeless-scraping-browser live

# Get live preview URL for specific session
scrapeless-scraping-browser live abc123def456

# Returns live preview URL for browser viewing
# Open this URL in your browser to view the live session

Timeouts and Slow Pages

For slow websites, use explicit waits:

# Wait for network activity to settle
scrapeless-scraping-browser wait --load networkidle

# Wait for specific element
scrapeless-scraping-browser wait @e1

# Wait for URL pattern
scrapeless-scraping-browser wait --url "**/dashboard"

# Wait fixed duration (milliseconds)
scrapeless-scraping-browser wait 5000

Error Handling

Common errors and solutions:

Authentication Error:

# Make sure API token is set
scrapeless-scraping-browser config set apiKey your_token
# Or use environment variable
export SCRAPELESS_API_KEY=your_token

Session Not Found:

# Session may have expired, create new one
scrapeless-scraping-browser open https://example.com

Element Not Found:

# Re-snapshot to get fresh refs
scrapeless-scraping-browser snapshot -i

Timeout:

# Increase session timeout (in seconds)
scrapeless-scraping-browser config set sessionTtl 600
scrapeless-scraping-browser open https://example.com

Debugging

Enable debug mode for detailed logs:

scrapeless-scraping-browser config set debug true
scrapeless-scraping-browser open https://example.com

Or use --debug flag:

scrapeless-scraping-browser --debug open https://example.com

Configuration Options

Set configuration using:

scrapeless-scraping-browser config set <key> <value>

Or use environment variable for API key only:

export SCRAPELESS_API_KEY=your_token

Key Differences from Local Browsers

1. Cloud-based: Runs on Scrapeless infrastructure, not locally
2. Residential Proxies: Built-in support for residential proxy rotation
3. Anti-detection: Automatic browser fingerprinting and stealth features
4. Session Recording: Optional recording of browser sessions
5. No Installation: No need to install Chrome/Chromium locally
6. Scalable: Run multiple sessions in parallel

Limitations

• Profile management is not currently supported
• Browser extensions are not currently supported
• Requires active internet connection
• Requires valid Scrapeless API token

Best Practices

1. Always set API token before running commands (via config or env var)
2. Let automatic session management work - the CLI will reuse sessions intelligently
3. Use --session-id only when you need parallel workflows
4. Close sessions when done to avoid charges
5. Use config file for persistent settings instead of environment variables
6. Enable recording for debugging complex flows
7. Re-snapshot after page changes
8. Use --json for programmatic access
9. Set session timeout appropriately for your use case (in seconds)

Updates

Check for and install updates:

# Check version
scrapeless-scraping-browser version

# Update via npm
npm update -g scrapeless-scraping-browser-skills

Support

• Documentation: https://docs.scrapeless.com
• API Reference: https://api.scrapeless.com/docs
• GitHub Issues: https://github.com/scrapeless-ai/scrapeless-agent-browser/issues