# Scraping HTML To Markdown

> >- Use when the user wants a single page rendered as clean Markdown plus structured metadata. Covers `crawlberg scrape URL`, JSON vs Markdown output, what metadata is returned, and how to handle JS-heavy pages.

- Skill: `thedixitjain/scraping-html-to-markdown` (Agent Skill)
- Install (CLI): `npx skillmds add thedixitjain/scraping-html-to-markdown`
- Raw SKILL.md: https://api.skillmd.com/api/skills/thedixitjain/scraping-html-to-markdown/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Web & Frontend
- Author: thedixitjain (https://skillmd.com/u/thedixitjain)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/thedixitjain/scraping-html-to-markdown

---



<!--
AI-RULEZ :: GENERATED FILE — DO NOT EDIT
Content-Hash: blake3:d4aa39402d49ad2eada44eea60c4a03a463b56134d3b869b5119f23d06144211
Source-Hash: blake3:b5689383a3914da8e3cc6ad3614356ae7f98c90cf99cb1b4b47be38456ff7a7c
Schema-Version: v1
-->

# Scraping HTML to Markdown

`crawlberg scrape <url>` is the right tool when the user has a single
page in mind. It returns Markdown plus a full structured payload (metadata,
links, images, JSON-LD, HTTP response info).

## Quick recipe

```bash
crawlberg scrape https://example.com/article --format markdown
```

JSON form (default) when downstream needs metadata:

```bash
crawlberg scrape https://example.com/article --format json
```

## Flag surface

| Flag                    | Default | Purpose                                                |
| ----------------------- | ------- | ------------------------------------------------------ |
| `--format`              | `json`  | `json` or `markdown`.                                  |
| `--timeout`             | `30000` | Per-request timeout in ms.                             |
| `--proxy`               | —       | HTTP, HTTPS, or SOCKS5 proxy URL.                      |
| `--user-agent`          | —       | Override request UA.                                   |
| `--respect-robots-txt`  | off     | Honour `robots.txt`.                                   |
| `--browser-mode`        | `auto`  | `auto`, `always`, `never` — see headless-fallback skill. |
| `--browser-endpoint`    | —       | External CDP `ws://` URL.                              |
| `--config`              | —       | Inline JSON or `@file.json` for full `CrawlConfig`.    |

## Output shape

### Markdown mode

Prints the rendered Markdown only. Use when piping to a file the user will
read, or when the result becomes LLM context downstream.

### JSON mode

Top-level `PageResult` with:

- `url`, `final_url` (after redirects), `status_code`.
- `markdown`: `{ content, fit_content, warnings }` — `fit_content` is a
  pruned LLM-optimised variant.
- `metadata`: Open Graph, Twitter Card, Dublin Core, article tags, JSON-LD,
  headings (H1–H6), feeds, favicons, hreflang.
- `links`: arrays for `Internal`, `External`, `Anchor`, and `Document`.
- `images`: `<img>`, `<picture>`, `srcset`, `og:image`.
- `tables`: structured table data preserved separately from Markdown.
- `response`: HTTP headers, content type, charset, body size.

Read `result.markdown.content` for the Markdown string when scripting.

## Common pitfalls

### Empty or stub content

Static fetch returned a JS shell. Symptoms in JSON output:

- `markdown.content` is short or only contains nav/footer chrome.
- `markdown.warnings` mentions JS-render-required.
- `metadata.headings` is empty when the page clearly has headings.

Re-run with `--browser-mode always` and see the headless-fallback skill.

### WAF block

`Auto` mode detects 8 WAF vendors and retries through headless Chrome
automatically. If you forced `--browser-mode never`, the WAF response will
fall through. Check `response.status_code` — 403/406/503 with WAF headers
(`server: cloudflare`, `x-amz-cf-id`, etc.) is the giveaway.

### Robots.txt blocking the fetch

If `--respect-robots-txt` is set and the path is disallowed, the scrape
returns an error rather than partial content. Drop the flag only on hosts
you own or have authorisation for.

### Wrong charset

Most pages declare UTF-8. Pages that lie about their charset can surface as
mojibake in `markdown.content`. crawlberg exposes no encoding-override option
(there is no `force_encoding`/charset field in `CrawlConfig`, and `--config`
rejects unknown keys), so an incorrectly declared charset is a server-side defect —
re-fetch the raw bytes and transcode them downstream if you hit it.

## Examples

### Scrape an article for downstream LLM context

```bash
crawlberg scrape https://blog.example.com/post-123 --format markdown \
  > /tmp/article.md
```

### Scrape with proxy and custom UA

```bash
crawlberg scrape https://example.com \
  --proxy http://proxy.internal:3128 \
  --user-agent "crawlberg (research@example.com)" \
  --format json
```

### Extract just the OG metadata

```bash
crawlberg scrape https://example.com --format json \
  | jq '.metadata | {title: .og.title, description: .og.description, image: .og.image}'
```

## When to reach for crawl or interact instead

- The user wants the whole site, not one page → `crawling-a-site` skill.
- The user needs to click, type, or scroll before extracting → use
  `crawlberg interact` with the action list.
- The user only wants the list of URLs → `crawlberg map`.

---

**Source:** [`hashgraph-online/awesome-codex-plugins`](https://github.com/hashgraph-online/awesome-codex-plugins) → `plugins/kreuzberg-dev/plugins/plugins/crawlberg/skills/scraping-html-to-markdown/SKILL.md`