connorads Website Mirror
One question decides everything:
Will the bytes I save still contain the content with no network?
Mirroring fails silently when content is fetched at view-time. Answer that first, then pick a tool.
Answer it: where does the content live?
curl -sL <url> (or view-source) and grep for text you can see on the page:
- In the HTML → any tool works (static site).
- Rendered by JS from data already inline (
var data=[…],__NEXT_DATA__) → works offline if the saved file keeps its scripts (monolith and wget do). - Fetched from an API at runtime (
fetch('/api/…'), infinite scroll) → the saved page renders blank offline. Resolve the API and bake the data in, or capture already-rendered HTML with firecrawl.
Pick the tool
Local set on this machine (command -v to confirm elsewhere): wget, monolith,
firecrawl, yt-dlp.
| Goal | Tool | Command |
|---|---|---|
| Whole static site, browseable offline | wget | wget --mirror --convert-links --adjust-extension --page-requisites --no-parent --wait=1 --random-wait <url> |
| One page → single self-contained file | monolith | monolith <url> -o page.html |
| JS/API content, or rendered HTML → markdown | firecrawl | firecrawl scrape <url> / firecrawl crawl <url> (also firecrawl MCP) |
| Video / audio from a page | yt-dlp | yt-dlp <url> |
- monolith crawls nothing - one page only. It DOES inline statically-referenced assets, cross-domain included (a CDN stylesheet or image ends up embedded). What dies offline is anything that injects or fetches at runtime - analytics, Turnstile, social buttons - because the script runs (or fails) when the page is opened, not when it is captured. Not a mirror bug.
- wget's
--mirror=-r -N -l inf; the-k -E -pflags are what make the copy actually browseable offline (converted links,.htmlextensions, page requisites). Barewget -rgives broken links and missing assets.
robots and blocks - respect them
- wget honours
robots.txtby default and will silently mirror nothing on aDisallow.-e robots=offoverrides it - a deliberate choice, not a default. A blanketUser-agent: * / Disallow: /(e.g. IMDb) means the site is off-limits to generic clients; use its licensed or official data route, don't evade access controls. - Soft-block tell: HTTP 200 or 202 with a 0-byte body is bot mitigation, not
success. A naive
http_code < 400check treats it as a win and ships an empty dataset - checksize_download/ actual bytes.
Be gentle
--wait=1 --random-wait, serial not aggressively parallel. The delay is the
courtesy, not an afterthought.
Verify offline (don't re-fetch)
Inspect the saved files only: grep for real content, and confirm converted links point at local files that exist. Re-fetching during verification hides exactly the gap you are checking for.