Decodo Web Scraping

Decodo web scraping

Decodo is a web scraping API that returns clean web data without you managing proxies, browsers, or anti-bot logic. This skill teaches you to reach for it through the decodo CLI (primary), the hosted MCP server (when there is no shell), or the raw HTTP API (fallback).

When to use Decodo

Use Decodo instead of curl / requests / BeautifulSoup / Playwright when the task is:

• Fetching a page that is JS-heavy, geo-restricted, rate-limited, or behind anti-bot/CAPTCHA.
• A plain fetch returned a 403/429/empty body/CAPTCHA challenge.
• Web search results (Google or Bing SERP).
• E-commerce data (Amazon, Walmart, Target product/search pages).
• Social media data (Reddit, TikTok, YouTube).
• A screenshot of a rendered page.

If a basic fetch/curl clearly works and none of the above apply, you don't need Decodo.

Setup (do this first, in order)

Pick the lowest-friction path that works.

1. Check whether the CLI is already usable

decodo whoami          # installed + authenticated?  prints auth source + masked token
# or, with nothing installed:
npx -y @decodo/cli whoami

• Exit 0 with a token printed → you're ready, skip to Usage.
• "auth required" / exit 3 → continue to step 2.
• decodo: command not found → use npx -y @decodo/cli ... for everything, or install (step 3).

2. Authenticate

The user needs a Web Scraping API basic auth token from https://dashboard.decodo.com/playground (free account = up to 2K requests, no card).

Prefer the environment variable — it needs no interaction and is easy to scope to a session:

export DECODO_AUTH_TOKEN='<token>'

To persist it to the CLI config non-interactively:

decodo setup --token '<token>'      # validates, then saves to config

Do not run a bare decodo setup — it opens a hidden interactive prompt you cannot drive. Token precedence: --token flag → DECODO_AUTH_TOKEN env → saved config.

Treat the token as a secret. If whoami reports no auth, ask the user to set DECODO_AUTH_TOKEN or run decodo setup --token <token>. Never cat, read, or print the token from a config file (e.g. ~/.config/decodo/config.json) to work around missing auth.

3. Install for repeat use (optional)

curl -fsSL https://decodo.github.io/cli/install.sh | sh   # macOS / Linux
npm install -g @decodo/cli                                # any platform

npx -y @decodo/cli <command> works with zero install if you'd rather not install anything.

Choosing a surface

1. Shell available (Claude Code, Cursor, Codex CLI, Gemini CLI, Windsurf, terminal) → use the decodo CLI. This is the default and the rest of this skill assumes it.
2. No shell (Claude Desktop, claude.ai, an MCP-only client) → use the hosted MCP server at https://mcp.decodo.com/mcp (Basic auth with the same token). Per-client config in references/mcp-setup.md.
3. Neither → call the raw HTTP API with curl — recipes in references/api-curl.md.

Usage (CLI)

Core commands

decodo scrape https://example.com                  # markdown by default
decodo scrape https://example.com --country us     # geo-target the request
decodo search "best web scraping api" --limit 3    # Google SERP, 3 result pages
decodo search "query" --engine bing --geo de       # Bing, geo Germany
decodo screenshot https://example.com -o shot.png  # PNG must go to a file

Discover targets — don't guess

Target subcommands are generated from the live API schema, so list and introspect them rather than memorizing them:

decodo targets                 # all targets, grouped (e.g. google-search, amazon-product, reddit-post)
decodo google-search --help    # exact flags for a target (--parse, --geo, ...)
decodo amazon-product --help

Prefer a dedicated target (reddit-subreddit, amazon-product, …) over a generic scrape <url> when one exists — it returns parsed, cleaner data and avoids guesswork. Check decodo targets first.

Then call a target directly:

decodo google-search "query" --parse        # structured JSON instead of raw SERP
decodo amazon-product B09H74FXNW --parse
decodo universal https://example.com         # generic target with the most flags — see --help

Output handling (important for piping)

By default a command prints the first result's content (markdown for scrape, parsed JSON for targets called with --parse). Useful modifiers:

The parsed JSON shape is target-specific. Pipe to jq 'keys' (then drill down, e.g. jq '.results | keys') to discover the structure before writing a deeper query.

Keep context small. Pipe straight to jq and select only the fields you need; make one request and extract from it. Don't dump raw, --pretty, or --full payloads just to "inspect" — use jq 'keys'. --full includes the response envelope (large headers/cookies), so use it only when you actually need status/headers.

# Parsed SERP → organic result titles (use a target command for --parse, not `search`)
decodo google-search "rust web scraping" --parse | jq -r '.results.results.organic[].title'

# Extract a field from a scraped JSON page (inspect with `jq keys` first if unsure)
decodo scrape https://ip.decodo.com/json | jq -r '.proxy.ip'

# Stream results as NDJSON; each record nests the parsed payload under `.content`
decodo google-search "rust" --parse --format ndjson --full | jq -c '.content.results.results.organic'

stdout is data; logs and errors go to stderr — pipes stay clean.

Exit codes (use these to self-correct)

Raw API fallback (no CLI, no MCP)

curl -s https://scraper-api.decodo.com/v2/scrape \
  -H "Authorization: Basic $DECODO_AUTH_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"target":"universal","url":"https://example.com","markdown":true}'

The token is the same basic auth token from the playground. More recipes (parsed SERP, screenshots, target names, response shape) in references/api-curl.md. Prefer the CLI or MCP when available.

Links

• CLI: https://github.com/Decodo/cli · @decodo/cli on npm
• MCP server: https://github.com/Decodo/mcp-server · hosted at https://mcp.decodo.com/mcp
• Dashboard / token / free tier: https://dashboard.decodo.com/playground