# Url Scrape

> Use when the user wants to scrape any URL or web page as Markdown/text through the local Frevana Chrome Extension-backed browser tool, including authenticated pages using the user's Chrome Extension login session.

- Skill: `finpeakinc/url-scrape` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds@latest add finpeakinc/url-scrape`
- Raw SKILL.md: https://api.skillmd.com/api/skills/finpeakinc/url-scrape/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Docs & Writing
- Author: finpeakinc (https://skillmd.com/u/finpeakinc)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/finpeakinc/url-scrape

---


# URL Scrape

Scrape any URL through the local Frevana daemon and Chrome Extension session.

## Purpose

This skill is for **scraping a web page URL** using the Chrome Extension-backed Frevana MCP tool `frevana_scrape`.

Inputs:

- `url` (required) - absolute URL to scrape
- `provider` (optional) - defaults to `url` for clean Markdown extraction via Readability.js
- `timeout` (optional) - Frevana tool timeout in milliseconds, default comes from the tool

Output:

- scraped page content, usually clean Markdown with title, author, content, and links

Summarize the scraped content unless the user asks for the raw scrape output.

## What This Skill Needs

- user-provided absolute `url`
- bundled `scripts/setup.sh` wrapper, which downloads and runs the latest official Frevana setup script
- Frevana local daemon running after setup, default port `12306`
- Chrome connected through the Frevana Chrome Extension
- `curl`
- `bash`
- `python3`

This is a Chrome Extension skill. It uses the local daemon and Chrome Extension session.

## Execution Order

Use this flow:

1. Confirm the user has provided an absolute URL with `http://` or `https://`.
2. Prefer the script over ad hoc `frevana call` commands.
3. Default `provider` to `url` when the user does not specify it.
4. Do not invent optional timeout values.
5. Let the script run bundled `scripts/setup.sh` before every Frevana tool call.
6. If setup reports Chrome disconnected, stop and tell the user to open Chrome, connect the Frevana extension, and retry.
7. Run the URL scrape only after setup succeeds.
8. If the scrape returns login/auth content or empty content, tell the user they may need to log in to that site in Chrome.
9. Return a concise summary, or the raw scraped content if requested.
10. When useful, save the output to a file.

## Commands

### Basic URL scrape

```bash
bash <skill-path>/scripts/scrape_url.sh \
  --url "https://example.com"
```

### Explicit provider and timeout

```bash
bash <skill-path>/scripts/scrape_url.sh \
  --url "https://example.com" \
  --provider url \
  --timeout 60000
```

### Save output to file

```bash
bash <skill-path>/scripts/scrape_url.sh \
  --url "https://example.com" \
  --output ./out/url-scrape-result.md
```

## Fixed Tool Call Shape

The script calls:

```bash
frevana call frevana_scrape '<json_args>'
```

The JSON arguments use this shape, omitting optional fields that were not provided:

```json
{
  "url": "https://example.com",
  "provider": "url",
  "timeout": 60000
}
```

Always send `provider`; default it to `url` when the user does not specify a value.
Do not pass unsupported fields to `frevana_scrape`.

## Output

- Success: the script prints the Frevana scrape result to stdout
- With `--output`: the same result is also written to the specified file path
- Failure: the script prints the Frevana error or preflight failure to stderr and exits non-zero

## Notes

- Require `--url`.
- URL must start with `http://` or `https://`.
- `--provider` defaults to `url`.
- `--timeout` must be a positive integer when provided.
- The script runs bundled `scripts/setup.sh` before every scrape, matching the original Frevana skill flow.
- `scripts/setup.sh` downloads and executes the latest official setup script from `https://raw.githubusercontent.com/FinpeakInc/frevana-cli-releases/refs/heads/main/skills/frevana/scripts/setup.sh`.
- If `frevana` is missing, the official setup script installs the Frevana binary before starting/checking the daemon.

## Example Prompts

### 中文

- "抓取 https://example.com 的内容"
- "把这个网页 scrape 成 Markdown"
- "用 Frevana 抓取这个登录后的页面，并保存原始输出"

### English

- "Scrape https://example.com"
- "Extract this web page as Markdown"
- "Scrape this authenticated page through Chrome and save the raw output"